Oct 30 00:01:56.943609 ip-10-0-143-120 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 00:01:56.943620 ip-10-0-143-120 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 00:01:56.943626 ip-10-0-143-120 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 00:01:56.943882 ip-10-0-143-120 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 00:02:07.192989 ip-10-0-143-120 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 00:02:07.193006 ip-10-0-143-120 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot d82744948ca64261b14ddf995affed70 -- Oct 30 00:04:02.949179 ip-10-0-143-120 systemd[1]: Starting Kubernetes Kubelet... Oct 30 00:04:03.358881 ip-10-0-143-120 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:04:03.358881 ip-10-0-143-120 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 00:04:03.358881 ip-10-0-143-120 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:04:03.358881 ip-10-0-143-120 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:04:03.365004 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.364891 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:04:03.365004 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365000 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:04:03.365004 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365004 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:04:03.365004 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365008 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:04:03.365004 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365011 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:04:03.365149 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365014 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:04:03.365149 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365017 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:04:03.365149 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365019 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:04:03.365149 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365022 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:04:03.365149 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365025 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:04:03.365149 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365028 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:04:03.365149 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365032 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:04:03.365149 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365035 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:04:03.365149 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365037 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:04:03.365149 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365041 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:04:03.365149 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365043 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:04:03.365149 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365046 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:04:03.365149 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365049 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:04:03.365149 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365052 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:04:03.365149 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365055 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:04:03.365149 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365058 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:04:03.365149 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365073 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:04:03.365149 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365076 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:04:03.365149 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365079 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:04:03.365149 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365082 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:04:03.365603 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365084 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:04:03.365603 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365087 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:04:03.365603 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365089 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:04:03.365603 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365092 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:04:03.365603 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365094 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:04:03.365603 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365097 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:04:03.365603 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365099 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:04:03.365603 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365102 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:04:03.365603 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365106 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:04:03.365603 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365111 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:04:03.365603 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365114 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:04:03.365603 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365117 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:04:03.365603 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365120 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:04:03.365603 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365123 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:04:03.365603 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365125 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:04:03.365603 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365128 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:04:03.365603 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365131 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:04:03.365603 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365133 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:04:03.365603 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365136 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:04:03.366068 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365139 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:04:03.366068 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365141 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:04:03.366068 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365144 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:04:03.366068 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365147 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:04:03.366068 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365149 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:04:03.366068 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365152 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:04:03.366068 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365154 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:04:03.366068 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365157 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:04:03.366068 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365160 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:04:03.366068 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365163 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:04:03.366068 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365169 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:04:03.366068 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365172 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:04:03.366068 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365175 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:04:03.366068 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365178 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:04:03.366068 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365180 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:04:03.366068 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365183 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:04:03.366068 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365185 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:04:03.366068 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365188 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:04:03.366068 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365191 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:04:03.366068 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365193 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:04:03.366546 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365196 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:04:03.366546 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365198 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:04:03.366546 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365201 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:04:03.366546 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365204 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:04:03.366546 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365207 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:04:03.366546 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365211 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:04:03.366546 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365213 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:04:03.366546 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365216 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:04:03.366546 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365218 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:04:03.366546 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365221 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:04:03.366546 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365225 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:04:03.366546 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365227 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:04:03.366546 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365230 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:04:03.366546 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365232 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:04:03.366546 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365235 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:04:03.366546 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365237 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:04:03.366546 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365240 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:04:03.366546 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365243 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:04:03.366546 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365246 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:04:03.366546 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365248 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:04:03.367030 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365251 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:04:03.367030 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365254 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:04:03.367030 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365256 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:04:03.367030 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365259 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:04:03.367030 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365262 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:04:03.367030 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365265 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:04:03.367030 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365267 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:04:03.367030 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365270 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:04:03.367030 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365272 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:04:03.367030 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365277 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:04:03.367030 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365282 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:04:03.367030 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365285 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:04:03.367030 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365289 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:04:03.367030 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365292 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:04:03.367030 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365295 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:04:03.367030 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365298 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:04:03.367030 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365301 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:04:03.367030 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365304 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:04:03.367030 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365306 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:04:03.367505 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365309 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:04:03.367505 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365313 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:04:03.367505 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.365315 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:04:03.367505 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366131 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:04:03.367505 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366139 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:04:03.367505 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366142 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:04:03.367505 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366146 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:04:03.367505 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366148 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:04:03.367505 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366151 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:04:03.367505 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366154 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:04:03.367505 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366157 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:04:03.367505 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366159 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:04:03.367505 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366162 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:04:03.367505 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366165 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:04:03.367505 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366167 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:04:03.367505 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366170 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:04:03.367505 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366173 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:04:03.367505 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366177 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:04:03.367505 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366180 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:04:03.367505 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366182 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:04:03.367996 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366185 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:04:03.367996 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366188 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:04:03.367996 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366190 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:04:03.367996 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366193 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:04:03.367996 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366196 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:04:03.367996 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366198 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:04:03.367996 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366201 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:04:03.367996 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366204 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:04:03.367996 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366208 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:04:03.367996 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366210 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:04:03.367996 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366213 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:04:03.367996 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366215 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:04:03.367996 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366218 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:04:03.367996 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366220 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:04:03.367996 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366223 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:04:03.367996 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366226 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:04:03.367996 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366229 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:04:03.367996 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366232 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:04:03.367996 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366234 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:04:03.367996 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366237 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:04:03.368468 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366240 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:04:03.368468 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366242 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:04:03.368468 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366245 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:04:03.368468 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366247 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:04:03.368468 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366250 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:04:03.368468 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366253 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:04:03.368468 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366255 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:04:03.368468 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366258 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:04:03.368468 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366262 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:04:03.368468 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366266 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:04:03.368468 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366270 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:04:03.368468 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366273 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:04:03.368468 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366276 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:04:03.368468 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366279 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:04:03.368468 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366282 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:04:03.368468 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366284 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:04:03.368468 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366287 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:04:03.368468 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366290 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:04:03.368468 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366293 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:04:03.369029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366295 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:04:03.369029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366298 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:04:03.369029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366300 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:04:03.369029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366303 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:04:03.369029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366305 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:04:03.369029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366308 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:04:03.369029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366319 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:04:03.369029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366323 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:04:03.369029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366325 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:04:03.369029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366328 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:04:03.369029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366331 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:04:03.369029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366333 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:04:03.369029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366336 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:04:03.369029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366339 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:04:03.369029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366341 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:04:03.369029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366345 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:04:03.369029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366348 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:04:03.369029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366350 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:04:03.369029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366353 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:04:03.369471 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366356 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:04:03.369471 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366359 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:04:03.369471 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366361 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:04:03.369471 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366364 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:04:03.369471 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366367 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:04:03.369471 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366370 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:04:03.369471 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366373 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:04:03.369471 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366376 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:04:03.369471 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366378 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:04:03.369471 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366381 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:04:03.369471 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366383 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:04:03.369471 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366386 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:04:03.369471 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366389 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:04:03.369471 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366391 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:04:03.369471 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366394 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:04:03.369471 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366397 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:04:03.369471 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366399 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:04:03.369471 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366402 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:04:03.369471 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366405 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:04:03.369471 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366408 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:04:03.370029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366411 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:04:03.370029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366413 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:04:03.370029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366416 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:04:03.370029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366419 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:04:03.370029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366422 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:04:03.370029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366424 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:04:03.370029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366426 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:04:03.370029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366429 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:04:03.370029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366434 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:04:03.370029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366438 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:04:03.370029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.366440 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:04:03.370029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367005 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:04:03.370029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367012 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:04:03.370029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367015 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:04:03.370029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367018 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:04:03.370029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367021 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:04:03.370029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367024 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:04:03.370029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367027 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:04:03.370029 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367030 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:04:03.370775 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367033 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:04:03.370775 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367036 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:04:03.370775 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367039 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:04:03.370775 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367042 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:04:03.370775 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367044 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:04:03.370775 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367048 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:04:03.370775 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367051 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:04:03.370775 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367053 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:04:03.370775 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367056 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:04:03.370775 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367059 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:04:03.370775 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367062 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:04:03.370775 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367065 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:04:03.370775 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367067 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:04:03.370775 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367070 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:04:03.370775 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367073 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:04:03.370775 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367076 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:04:03.370775 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367080 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:04:03.370775 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367083 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:04:03.370775 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367086 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:04:03.371377 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367089 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:04:03.371377 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367091 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:04:03.371377 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367094 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:04:03.371377 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367096 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:04:03.371377 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367099 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:04:03.371377 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367102 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:04:03.371377 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367105 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:04:03.371377 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367108 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:04:03.371377 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367110 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:04:03.371377 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367113 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:04:03.371377 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367115 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:04:03.371377 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367118 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:04:03.371377 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367121 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:04:03.371377 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367124 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:04:03.371377 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367126 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:04:03.371377 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367129 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:04:03.371377 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367131 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:04:03.371377 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367134 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:04:03.371377 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367137 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:04:03.371377 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367140 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:04:03.371853 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367142 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:04:03.371853 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367145 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:04:03.371853 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367148 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:04:03.371853 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367150 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:04:03.371853 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367153 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:04:03.371853 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367155 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:04:03.371853 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367158 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:04:03.371853 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367160 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:04:03.371853 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367163 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:04:03.371853 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367165 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:04:03.371853 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367168 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:04:03.371853 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367170 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:04:03.371853 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367173 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:04:03.371853 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367175 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:04:03.371853 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367178 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:04:03.371853 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367180 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:04:03.371853 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367183 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:04:03.371853 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367186 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:04:03.371853 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367189 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:04:03.371853 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367192 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:04:03.372331 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367194 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:04:03.372331 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367197 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:04:03.372331 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367199 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:04:03.372331 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367202 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:04:03.372331 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367206 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:04:03.372331 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367210 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:04:03.372331 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367213 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:04:03.372331 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367216 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:04:03.372331 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367218 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:04:03.372331 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367221 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:04:03.372331 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367224 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:04:03.372331 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367226 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:04:03.372331 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367229 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:04:03.372331 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367231 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:04:03.372331 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367234 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:04:03.372331 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367237 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:04:03.372331 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367240 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:04:03.372331 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367242 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:04:03.372331 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367245 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:04:03.372804 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367248 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:04:03.372804 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367251 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:04:03.372804 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367253 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:04:03.372804 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367256 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:04:03.372804 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367258 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:04:03.372804 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367260 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:04:03.372804 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367263 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:04:03.372804 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367265 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:04:03.372804 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367268 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:04:03.372804 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367271 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:04:03.372804 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367273 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:04:03.372804 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367276 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:04:03.372804 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367279 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:04:03.372804 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367281 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:04:03.372804 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367284 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:04:03.372804 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367287 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:04:03.372804 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367289 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:04:03.372804 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367292 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:04:03.372804 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367294 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:04:03.372804 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.367300 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:04:03.373302 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369214 2580 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 00:04:03.373302 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369225 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 00:04:03.373302 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369231 2580 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 00:04:03.373302 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369236 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 00:04:03.373302 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369241 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 00:04:03.373302 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369244 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 00:04:03.373302 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369248 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 00:04:03.373302 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369253 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 00:04:03.373302 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369256 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 00:04:03.373302 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369260 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 00:04:03.373302 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369263 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 00:04:03.373302 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369267 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 00:04:03.373302 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369270 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 00:04:03.373302 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369274 2580 flags.go:64] FLAG: --cgroup-root="" Oct 30 00:04:03.373302 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369276 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 00:04:03.373302 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369280 2580 flags.go:64] FLAG: --client-ca-file="" Oct 30 00:04:03.373302 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369282 2580 flags.go:64] FLAG: --cloud-provider="external" Oct 30 00:04:03.373302 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369286 2580 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 00:04:03.373302 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369290 2580 flags.go:64] FLAG: --cluster-domain="" Oct 30 00:04:03.373302 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369293 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 00:04:03.373302 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369296 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 00:04:03.373302 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369299 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 00:04:03.373302 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369303 2580 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 00:04:03.373302 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369307 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 00:04:03.373849 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369310 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 00:04:03.373849 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369314 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 00:04:03.373849 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369317 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 00:04:03.373849 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369320 2580 flags.go:64] FLAG: --contention-profiling="false" Oct 30 00:04:03.373849 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369324 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 00:04:03.373849 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369326 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 00:04:03.373849 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369330 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 00:04:03.373849 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369334 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 00:04:03.373849 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369338 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 00:04:03.373849 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369342 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 00:04:03.373849 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369345 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 00:04:03.373849 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369348 2580 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 00:04:03.373849 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369351 2580 flags.go:64] FLAG: --enable-server="true" Oct 30 00:04:03.373849 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369354 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 00:04:03.373849 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369358 2580 flags.go:64] FLAG: --event-burst="100" Oct 30 00:04:03.373849 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369361 2580 flags.go:64] FLAG: --event-qps="50" Oct 30 00:04:03.373849 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369365 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 00:04:03.373849 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369368 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 00:04:03.373849 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369371 2580 flags.go:64] FLAG: --eviction-hard="" Oct 30 00:04:03.373849 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369375 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 00:04:03.373849 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369378 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 00:04:03.373849 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369381 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 00:04:03.373849 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369385 2580 flags.go:64] FLAG: --eviction-soft="" Oct 30 00:04:03.373849 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369388 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 00:04:03.373849 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369391 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 00:04:03.374483 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369395 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 00:04:03.374483 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369397 2580 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 00:04:03.374483 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369401 2580 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 00:04:03.374483 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369404 2580 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 00:04:03.374483 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369408 2580 flags.go:64] FLAG: --feature-gates="" Oct 30 00:04:03.374483 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369412 2580 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 00:04:03.374483 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369415 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 00:04:03.374483 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369418 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 00:04:03.374483 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369422 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 00:04:03.374483 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369425 2580 flags.go:64] FLAG: --healthz-port="10248" Oct 30 00:04:03.374483 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369428 2580 flags.go:64] FLAG: --help="false" Oct 30 00:04:03.374483 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369431 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-143-120.ec2.internal" Oct 30 00:04:03.374483 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369434 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 00:04:03.374483 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369440 2580 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 00:04:03.374483 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369444 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 00:04:03.374483 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369447 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 00:04:03.374483 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369451 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 00:04:03.374483 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369454 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 00:04:03.374483 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369457 2580 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 00:04:03.374483 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369460 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 00:04:03.374483 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369463 2580 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 00:04:03.374483 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369466 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 00:04:03.374483 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369470 2580 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 00:04:03.374483 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369473 2580 flags.go:64] FLAG: --kube-reserved="" Oct 30 00:04:03.375067 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369477 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 00:04:03.375067 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369480 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 00:04:03.375067 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369483 2580 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 00:04:03.375067 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369486 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 00:04:03.375067 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369489 2580 flags.go:64] FLAG: --lock-file="" Oct 30 00:04:03.375067 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369492 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 00:04:03.375067 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369496 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 00:04:03.375067 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369499 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 00:04:03.375067 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369505 2580 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 00:04:03.375067 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369508 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 00:04:03.375067 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369511 2580 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 00:04:03.375067 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369515 2580 flags.go:64] FLAG: --logging-format="text" Oct 30 00:04:03.375067 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369518 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 00:04:03.375067 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369521 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 00:04:03.375067 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369524 2580 flags.go:64] FLAG: --manifest-url="" Oct 30 00:04:03.375067 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369527 2580 flags.go:64] FLAG: --manifest-url-header="" Oct 30 00:04:03.375067 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369532 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 00:04:03.375067 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369535 2580 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 00:04:03.375067 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369539 2580 flags.go:64] FLAG: --max-pods="110" Oct 30 00:04:03.375067 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369542 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 00:04:03.375067 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369545 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 00:04:03.375067 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369549 2580 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 00:04:03.375067 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369552 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 00:04:03.375067 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369555 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 00:04:03.375067 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369558 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 00:04:03.375663 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369562 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 00:04:03.375663 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369570 2580 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 00:04:03.375663 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369573 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 00:04:03.375663 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369576 2580 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 00:04:03.375663 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369579 2580 flags.go:64] FLAG: --pod-cidr="" Oct 30 00:04:03.375663 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369582 2580 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 00:04:03.375663 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369585 2580 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 00:04:03.375663 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369588 2580 flags.go:64] FLAG: --pods-per-core="0" Oct 30 00:04:03.375663 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369591 2580 flags.go:64] FLAG: --port="10250" Oct 30 00:04:03.375663 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369594 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 00:04:03.375663 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369597 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d2a8f9d9ac765af2" Oct 30 00:04:03.375663 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369600 2580 flags.go:64] FLAG: --qos-reserved="" Oct 30 00:04:03.375663 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369603 2580 flags.go:64] FLAG: --read-only-port="10255" Oct 30 00:04:03.375663 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369606 2580 flags.go:64] FLAG: --register-node="true" Oct 30 00:04:03.375663 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369609 2580 flags.go:64] FLAG: --register-with-taints="" Oct 30 00:04:03.375663 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369613 2580 flags.go:64] FLAG: --registry-burst="10" Oct 30 00:04:03.375663 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369616 2580 flags.go:64] FLAG: --registry-qps="5" Oct 30 00:04:03.375663 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369619 2580 flags.go:64] FLAG: --reserved-cpus="" Oct 30 00:04:03.375663 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369622 2580 flags.go:64] FLAG: --reserved-memory="" Oct 30 00:04:03.375663 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369626 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 00:04:03.375663 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369629 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 00:04:03.375663 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369632 2580 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 00:04:03.375663 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369635 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 00:04:03.375663 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369638 2580 flags.go:64] FLAG: --runonce="false" Oct 30 00:04:03.375663 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369641 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 00:04:03.375663 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369644 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 00:04:03.376303 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369647 2580 flags.go:64] FLAG: --seccomp-default="false" Oct 30 00:04:03.376303 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369650 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 00:04:03.376303 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369653 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 00:04:03.376303 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369657 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 00:04:03.376303 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369660 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 00:04:03.376303 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369663 2580 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 00:04:03.376303 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369666 2580 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 00:04:03.376303 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369670 2580 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 00:04:03.376303 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369673 2580 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 00:04:03.376303 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369676 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 00:04:03.376303 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369679 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 00:04:03.376303 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369682 2580 flags.go:64] FLAG: --system-cgroups="" Oct 30 00:04:03.376303 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369685 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 00:04:03.376303 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369690 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 00:04:03.376303 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369693 2580 flags.go:64] FLAG: --tls-cert-file="" Oct 30 00:04:03.376303 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369696 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 00:04:03.376303 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369700 2580 flags.go:64] FLAG: --tls-min-version="" Oct 30 00:04:03.376303 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369702 2580 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 00:04:03.376303 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369705 2580 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 00:04:03.376303 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369709 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 00:04:03.376303 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369712 2580 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 00:04:03.376303 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369715 2580 flags.go:64] FLAG: --v="2" Oct 30 00:04:03.376303 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369719 2580 flags.go:64] FLAG: --version="false" Oct 30 00:04:03.376303 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369723 2580 flags.go:64] FLAG: --vmodule="" Oct 30 00:04:03.376303 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369728 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 00:04:03.376882 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.369731 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 00:04:03.376882 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371296 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:04:03.376882 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371311 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:04:03.376882 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371315 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:04:03.376882 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371318 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:04:03.376882 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371321 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:04:03.376882 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371324 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:04:03.376882 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371326 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:04:03.376882 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371329 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:04:03.376882 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371332 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:04:03.376882 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371335 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:04:03.376882 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371337 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:04:03.376882 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371340 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:04:03.376882 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371343 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:04:03.376882 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371345 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:04:03.376882 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371348 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:04:03.376882 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371350 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:04:03.376882 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371353 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:04:03.376882 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371355 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:04:03.376882 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371358 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:04:03.377391 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371362 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:04:03.377391 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371364 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:04:03.377391 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371368 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:04:03.377391 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371372 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:04:03.377391 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371377 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:04:03.377391 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371381 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:04:03.377391 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371386 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:04:03.377391 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371389 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:04:03.377391 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371392 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:04:03.377391 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371395 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:04:03.377391 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371398 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:04:03.377391 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371403 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:04:03.377391 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371406 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:04:03.377391 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371409 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:04:03.377391 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371412 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:04:03.377391 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371416 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:04:03.377391 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371419 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:04:03.377391 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371422 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:04:03.377391 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371425 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:04:03.377833 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371427 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:04:03.377833 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371430 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:04:03.377833 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371433 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:04:03.377833 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371436 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:04:03.377833 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371439 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:04:03.377833 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371441 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:04:03.377833 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371444 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:04:03.377833 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371447 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:04:03.377833 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371449 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:04:03.377833 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371452 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:04:03.377833 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371455 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:04:03.377833 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371457 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:04:03.377833 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371460 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:04:03.377833 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371463 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:04:03.377833 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371466 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:04:03.377833 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371468 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:04:03.377833 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371471 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:04:03.377833 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371474 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:04:03.377833 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371477 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:04:03.377833 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371479 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:04:03.378351 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371483 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:04:03.378351 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371485 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:04:03.378351 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371488 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:04:03.378351 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371490 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:04:03.378351 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371494 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:04:03.378351 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371497 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:04:03.378351 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371500 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:04:03.378351 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371503 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:04:03.378351 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371505 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:04:03.378351 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371508 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:04:03.378351 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371511 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:04:03.378351 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371513 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:04:03.378351 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371516 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:04:03.378351 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371518 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:04:03.378351 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371521 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:04:03.378351 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371524 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:04:03.378351 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371526 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:04:03.378351 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371529 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:04:03.378351 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371532 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:04:03.378811 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371535 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:04:03.378811 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371538 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:04:03.378811 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371540 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:04:03.378811 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371543 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:04:03.378811 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371545 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:04:03.378811 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371548 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:04:03.378811 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371558 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:04:03.378811 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371561 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:04:03.378811 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371564 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:04:03.378811 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371567 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:04:03.378811 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371569 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:04:03.378811 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371572 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:04:03.378811 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371574 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:04:03.378811 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371577 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:04:03.378811 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371580 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:04:03.378811 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371582 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:04:03.378811 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371585 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:04:03.378811 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371589 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:04:03.378811 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371591 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:04:03.378811 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371594 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:04:03.379655 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371597 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:04:03.379655 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371600 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:04:03.379655 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371602 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:04:03.379655 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371605 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:04:03.379655 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371608 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:04:03.379655 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371610 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:04:03.379655 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371614 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:04:03.379655 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371618 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:04:03.379655 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371621 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:04:03.379655 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.371656 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:04:03.379655 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371699 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:04:03.379655 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371705 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:04:03.379655 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371709 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:04:03.379655 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371713 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:04:03.379655 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371716 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:04:03.379655 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371719 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:04:03.380366 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371721 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:04:03.380366 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371724 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:04:03.380366 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371727 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:04:03.380366 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371735 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:04:03.380366 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371738 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:04:03.380366 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371741 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:04:03.380366 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371743 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:04:03.380366 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371746 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:04:03.380366 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371749 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:04:03.380366 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371751 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:04:03.380366 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371754 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:04:03.380366 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371757 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:04:03.380366 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371759 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:04:03.380366 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371762 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:04:03.380366 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371766 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:04:03.380366 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371769 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:04:03.380366 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371771 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:04:03.380366 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371774 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:04:03.380366 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371776 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:04:03.381185 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371779 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:04:03.381185 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371782 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:04:03.381185 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371784 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:04:03.381185 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371787 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:04:03.381185 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371789 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:04:03.381185 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371792 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:04:03.381185 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371795 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:04:03.381185 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371797 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:04:03.381185 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371801 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:04:03.381185 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371804 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:04:03.381185 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371807 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:04:03.381185 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371809 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:04:03.381185 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371812 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:04:03.381185 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371814 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:04:03.381185 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371826 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:04:03.381185 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371829 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:04:03.381185 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371832 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:04:03.381185 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371835 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:04:03.381185 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371838 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:04:03.381185 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371841 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:04:03.381793 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371844 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:04:03.381793 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371847 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:04:03.381793 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371849 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:04:03.381793 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371852 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:04:03.381793 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371855 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:04:03.381793 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371857 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:04:03.381793 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371860 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:04:03.381793 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371864 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:04:03.381793 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371867 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:04:03.381793 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371870 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:04:03.381793 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371873 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:04:03.381793 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371875 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:04:03.381793 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371878 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:04:03.381793 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371880 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:04:03.381793 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371883 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:04:03.381793 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371886 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:04:03.381793 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371888 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:04:03.381793 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371891 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:04:03.381793 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371894 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:04:03.381793 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371896 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:04:03.382545 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371900 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:04:03.382545 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371903 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:04:03.382545 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371906 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:04:03.382545 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371924 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:04:03.382545 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371930 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:04:03.382545 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371934 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:04:03.382545 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371937 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:04:03.382545 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371940 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:04:03.382545 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371943 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:04:03.382545 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371946 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:04:03.382545 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371949 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:04:03.382545 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371952 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:04:03.382545 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371955 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:04:03.382545 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371957 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:04:03.382545 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371960 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:04:03.382545 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371963 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:04:03.382545 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371965 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:04:03.382545 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371968 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:04:03.382545 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371970 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:04:03.383443 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371974 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:04:03.383443 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371977 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:04:03.383443 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371979 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:04:03.383443 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371982 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:04:03.383443 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371984 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:04:03.383443 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371987 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:04:03.383443 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371990 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:04:03.383443 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371993 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:04:03.383443 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371995 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:04:03.383443 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.371998 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:04:03.383443 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.372002 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:04:03.383443 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.372006 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:04:03.383443 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.372009 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:04:03.383443 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.372013 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:04:03.383443 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.372016 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:04:03.383443 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.372020 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:04:03.383443 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.372023 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:04:03.383443 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.372025 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:04:03.383443 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.372028 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:04:03.384269 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.372030 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:04:03.384269 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.372033 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:04:03.384269 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.372036 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:04:03.384269 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.381412 2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 00:04:03.384269 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.381435 2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 00:04:03.384269 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.382116 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 00:04:03.384269 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.382131 2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 00:04:03.384269 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382222 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:04:03.384269 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382230 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:04:03.384269 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382237 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:04:03.384269 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382242 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:04:03.384269 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382247 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:04:03.384269 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382251 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:04:03.384269 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382256 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:04:03.384269 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382261 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:04:03.384269 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382265 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:04:03.384269 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382269 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:04:03.384269 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382274 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:04:03.384269 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382278 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:04:03.384774 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382283 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:04:03.384774 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382288 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:04:03.384774 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382293 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:04:03.384774 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382297 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:04:03.384774 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382301 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:04:03.384774 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382306 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:04:03.384774 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382318 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:04:03.384774 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382323 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:04:03.384774 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382327 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:04:03.384774 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382332 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:04:03.384774 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382336 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:04:03.384774 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382340 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:04:03.384774 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382345 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:04:03.384774 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382349 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:04:03.384774 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382354 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:04:03.384774 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382358 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:04:03.384774 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382363 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:04:03.384774 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382370 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:04:03.384774 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382375 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:04:03.384774 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382379 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:04:03.385526 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382384 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:04:03.385526 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382388 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:04:03.385526 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382393 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:04:03.385526 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382397 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:04:03.385526 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382402 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:04:03.385526 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382406 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:04:03.385526 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382411 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:04:03.385526 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382417 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:04:03.385526 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382424 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:04:03.385526 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382429 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:04:03.385526 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382433 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:04:03.385526 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382438 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:04:03.385526 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382442 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:04:03.385526 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382446 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:04:03.385526 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382451 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:04:03.385526 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382455 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:04:03.385526 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382462 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:04:03.385526 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382469 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:04:03.385526 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382474 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:04:03.386542 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382478 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:04:03.386542 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382482 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:04:03.386542 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382487 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:04:03.386542 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382491 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:04:03.386542 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382495 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:04:03.386542 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382500 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:04:03.386542 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382504 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:04:03.386542 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382508 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:04:03.386542 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382513 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:04:03.386542 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382544 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:04:03.386542 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382551 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:04:03.386542 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382557 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:04:03.386542 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382563 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:04:03.386542 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382568 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:04:03.386542 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382573 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:04:03.386542 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382578 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:04:03.386542 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382583 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:04:03.386542 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382587 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:04:03.386542 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382592 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:04:03.386542 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382596 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:04:03.387037 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382601 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:04:03.387037 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382605 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:04:03.387037 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382609 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:04:03.387037 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382614 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:04:03.387037 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382618 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:04:03.387037 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382622 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:04:03.387037 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382626 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:04:03.387037 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382631 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:04:03.387037 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382635 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:04:03.387037 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382639 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:04:03.387037 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382644 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:04:03.387037 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382648 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:04:03.387037 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382652 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:04:03.387037 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382656 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:04:03.387037 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382660 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:04:03.387037 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382665 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:04:03.387037 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382669 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:04:03.387037 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382674 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:04:03.387037 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382678 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:04:03.387627 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382682 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:04:03.387627 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382686 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:04:03.387627 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382691 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:04:03.387627 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382696 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:04:03.387627 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382700 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:04:03.387627 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382705 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:04:03.387627 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382710 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:04:03.387627 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382714 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:04:03.387627 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382718 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:04:03.387627 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382723 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:04:03.387627 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382727 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:04:03.387627 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382731 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:04:03.387627 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382735 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:04:03.387627 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382739 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:04:03.387627 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382743 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:04:03.387627 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.382748 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:04:03.388016 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.382793 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:04:03.388016 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383078 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:04:03.388016 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383089 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:04:03.388016 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383094 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:04:03.388016 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383099 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:04:03.388016 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383103 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:04:03.388016 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383108 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:04:03.388016 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383113 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:04:03.388016 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383117 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:04:03.388016 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383121 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:04:03.388016 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383126 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:04:03.388016 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383130 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:04:03.388016 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383134 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:04:03.388016 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383138 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:04:03.388016 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383142 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:04:03.388016 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383147 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:04:03.388384 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383152 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:04:03.388384 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383156 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:04:03.388384 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383161 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:04:03.388384 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383166 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:04:03.388384 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383170 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:04:03.388384 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383175 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:04:03.388384 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383180 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:04:03.388384 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383186 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:04:03.388384 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383193 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:04:03.388384 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383198 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:04:03.388384 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383203 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:04:03.388384 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383207 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:04:03.388384 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383212 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:04:03.388384 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383217 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:04:03.388384 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383221 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:04:03.388384 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383226 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:04:03.388384 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383230 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:04:03.388384 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383235 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:04:03.388384 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383239 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:04:03.388822 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383246 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:04:03.388822 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383251 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:04:03.388822 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383256 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:04:03.388822 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383260 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:04:03.388822 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383265 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:04:03.388822 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383269 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:04:03.388822 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383273 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:04:03.388822 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383278 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:04:03.388822 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383282 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:04:03.388822 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383286 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:04:03.388822 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383290 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:04:03.388822 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383295 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:04:03.388822 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383299 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:04:03.388822 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383304 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:04:03.388822 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383308 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:04:03.388822 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383312 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:04:03.388822 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383317 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:04:03.388822 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383321 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:04:03.388822 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383326 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:04:03.389335 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383330 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:04:03.389335 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383336 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:04:03.389335 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383340 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:04:03.389335 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383345 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:04:03.389335 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383349 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:04:03.389335 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383354 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:04:03.389335 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383358 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:04:03.389335 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383362 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:04:03.389335 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383367 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:04:03.389335 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383371 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:04:03.389335 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383375 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:04:03.389335 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383380 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:04:03.389335 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383384 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:04:03.389335 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383389 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:04:03.389335 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383393 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:04:03.389335 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383397 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:04:03.389335 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383401 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:04:03.389335 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383406 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:04:03.389335 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383410 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:04:03.389335 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383415 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:04:03.389815 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383419 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:04:03.389815 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383423 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:04:03.389815 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383427 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:04:03.389815 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383432 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:04:03.389815 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383436 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:04:03.389815 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383441 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:04:03.389815 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383445 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:04:03.389815 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383449 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:04:03.389815 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383453 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:04:03.389815 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383457 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:04:03.389815 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383461 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:04:03.389815 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383465 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:04:03.389815 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383469 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:04:03.389815 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383475 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:04:03.389815 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383480 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:04:03.389815 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383485 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:04:03.389815 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383489 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:04:03.389815 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383494 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:04:03.389815 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383498 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:04:03.389815 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383502 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:04:03.390393 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383506 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:04:03.390393 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383510 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:04:03.390393 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383514 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:04:03.390393 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383519 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:04:03.390393 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383523 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:04:03.390393 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383528 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:04:03.390393 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383532 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:04:03.390393 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383536 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:04:03.390393 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383541 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:04:03.390393 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383545 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:04:03.390393 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383549 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:04:03.390393 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383554 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:04:03.390393 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383558 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:04:03.390393 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.383604 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:04:03.390393 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383661 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:04:03.390393 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383667 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:04:03.390773 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383672 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:04:03.390773 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383677 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:04:03.390773 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383682 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:04:03.390773 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383686 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:04:03.390773 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383693 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:04:03.390773 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383699 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:04:03.390773 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383704 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:04:03.390773 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383709 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:04:03.390773 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383714 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:04:03.390773 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383718 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:04:03.390773 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383722 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:04:03.390773 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383728 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:04:03.390773 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383732 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:04:03.390773 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383737 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:04:03.390773 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383741 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:04:03.390773 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383746 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:04:03.390773 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383752 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:04:03.390773 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383758 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:04:03.390773 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383763 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:04:03.391520 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383768 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:04:03.391520 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383773 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:04:03.391520 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383777 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:04:03.391520 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383781 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:04:03.391520 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383786 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:04:03.391520 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383790 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:04:03.391520 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383794 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:04:03.391520 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383798 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:04:03.391520 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383803 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:04:03.391520 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383807 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:04:03.391520 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383811 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:04:03.391520 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383815 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:04:03.391520 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383819 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:04:03.391520 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383823 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:04:03.391520 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383828 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:04:03.391520 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383832 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:04:03.391520 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383837 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:04:03.391520 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383841 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:04:03.391520 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383845 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:04:03.392098 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383850 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:04:03.392098 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383854 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:04:03.392098 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383859 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:04:03.392098 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383863 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:04:03.392098 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383868 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:04:03.392098 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383872 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:04:03.392098 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383877 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:04:03.392098 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383882 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:04:03.392098 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383886 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:04:03.392098 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383890 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:04:03.392098 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383894 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:04:03.392098 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383899 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:04:03.392098 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383903 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:04:03.392098 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383925 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:04:03.392098 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383930 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:04:03.392098 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383934 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:04:03.392098 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383938 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:04:03.392098 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383943 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:04:03.392098 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383947 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:04:03.392098 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383951 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:04:03.392776 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383955 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:04:03.392776 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383959 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:04:03.392776 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383963 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:04:03.392776 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383967 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:04:03.392776 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383974 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:04:03.392776 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383979 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:04:03.392776 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383983 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:04:03.392776 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383989 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:04:03.392776 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383993 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:04:03.392776 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.383997 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:04:03.392776 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384003 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:04:03.392776 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384007 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:04:03.392776 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384011 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:04:03.392776 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384016 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:04:03.392776 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384021 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:04:03.392776 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384025 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:04:03.392776 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384029 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:04:03.392776 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384033 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:04:03.392776 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384037 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:04:03.392776 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384044 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:04:03.393594 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384048 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:04:03.393594 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384053 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:04:03.393594 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384057 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:04:03.393594 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384062 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:04:03.393594 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384066 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:04:03.393594 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384070 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:04:03.393594 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384075 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:04:03.393594 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384079 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:04:03.393594 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384084 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:04:03.393594 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384089 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:04:03.393594 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384093 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:04:03.393594 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384097 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:04:03.393594 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384101 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:04:03.393594 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384106 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:04:03.393594 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384110 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:04:03.393594 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384116 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:04:03.393594 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384123 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:04:03.393594 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384128 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:04:03.393594 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384132 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:04:03.394047 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384137 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:04:03.394047 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384141 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:04:03.394047 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384145 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:04:03.394047 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384149 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:04:03.394047 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384153 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:04:03.394047 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384158 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:04:03.394047 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:03.384162 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:04:03.394047 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.384259 2580 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 00:04:03.394047 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.387219 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 00:04:03.394047 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.388925 2580 server.go:1014] "Starting client certificate rotation" Oct 30 00:04:03.394047 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.389031 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 00:04:03.394047 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.389069 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 00:04:03.394047 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.392279 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 00:04:03.394875 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.394855 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 00:04:03.407079 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.407059 2580 log.go:25] "Validated CRI v1 runtime API" Oct 30 00:04:03.412289 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.412271 2580 log.go:25] "Validated CRI v1 image API" Oct 30 00:04:03.413567 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.413531 2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 00:04:03.418841 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.418812 2580 fs.go:135] Filesystem UUIDs: map[7827ee83-3ec9-436f-8d36-553a3f474bd7:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 90d22c60-f9c8-42ce-b35d-c0cf43c06df3:/dev/nvme0n1p4] Oct 30 00:04:03.418939 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.418839 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 00:04:03.424029 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.423878 2580 manager.go:217] Machine: {Timestamp:2026-10-30 00:04:03.422611721 +0000 UTC m=+0.365304554 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100617 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28a533568465876b15a59ff6315a42 SystemUUID:ec28a533-5684-6587-6b15-a59ff6315a42 BootID:d8274494-8ca6-4261-b14d-df995affed70 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c8:da:52:91 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c8:da:52:91 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:2c:ca:27:06:b0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 00:04:03.424029 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.424028 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 00:04:03.424131 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.424110 2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 00:04:03.425302 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.425272 2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 00:04:03.425442 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.425303 2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-120.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 00:04:03.425486 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.425444 2580 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 00:04:03.425486 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.425452 2580 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 00:04:03.425486 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.425484 2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 00:04:03.425561 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.425501 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 00:04:03.426593 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.426582 2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 00:04:03.430756 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.430743 2580 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 00:04:03.430882 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.430873 2580 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 00:04:03.431630 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.431620 2580 kubelet.go:493] "Attempting to sync node with API server" Oct 30 00:04:03.431663 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.431634 2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 00:04:03.431663 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.431645 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 00:04:03.431663 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.431654 2580 kubelet.go:405] "Adding apiserver pod source" Oct 30 00:04:03.431663 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.431663 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 00:04:03.432086 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.432061 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 00:04:03.434830 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.434811 2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 00:04:03.441588 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.441574 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 00:04:03.443327 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.443314 2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 00:04:03.443380 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.443345 2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 00:04:03.444785 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.444774 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 00:04:03.444824 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.444791 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 00:04:03.444824 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.444798 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 00:04:03.444824 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.444804 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 00:04:03.444824 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.444809 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 00:04:03.444824 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.444815 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 00:04:03.444824 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.444824 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 00:04:03.445006 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.444830 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 00:04:03.445006 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.444838 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 00:04:03.445006 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.444844 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 00:04:03.445006 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.444853 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 00:04:03.445006 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.444862 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 00:04:03.445768 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.445759 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 00:04:03.445768 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.445768 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 00:04:03.449634 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.449620 2580 server.go:1263] "Started kubelet" Oct 30 00:04:03.449741 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.449683 2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 00:04:03.449792 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.449732 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 00:04:03.449858 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.449799 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 00:04:03.450368 ip-10-0-143-120 systemd[1]: Started Kubernetes Kubelet. Oct 30 00:04:03.451087 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.451071 2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 00:04:03.451660 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.451644 2580 server.go:317] "Adding debug handlers to kubelet server" Oct 30 00:04:03.459206 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:03.459187 2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 00:04:03.459828 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.459808 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 00:04:03.460459 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.460440 2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 00:04:03.460542 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:03.460531 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-120.ec2.internal\" not found" Oct 30 00:04:03.461323 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.461309 2580 factory.go:55] Registering systemd factory Oct 30 00:04:03.461323 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.461325 2580 factory.go:223] Registration of the systemd container factory successfully Oct 30 00:04:03.461464 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.461438 2580 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 00:04:03.461464 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.461463 2580 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 00:04:03.461612 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.461440 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 00:04:03.461612 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.461516 2580 factory.go:153] Registering CRI-O factory Oct 30 00:04:03.461612 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.461525 2580 factory.go:223] Registration of the crio container factory successfully Oct 30 00:04:03.461612 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.461545 2580 reconstruct.go:97] "Volume reconstruction finished" Oct 30 00:04:03.461612 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.461552 2580 reconciler.go:29] "Reconciler: start to sync state" Oct 30 00:04:03.461612 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.461567 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 00:04:03.462428 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.462410 2580 factory.go:103] Registering Raw factory Oct 30 00:04:03.462428 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.462430 2580 manager.go:1199] Started watching for new ooms in manager Oct 30 00:04:03.463461 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.463445 2580 manager.go:319] Starting recovery of all containers Oct 30 00:04:03.470581 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.470556 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 00:04:03.474231 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.474213 2580 manager.go:324] Recovery completed Oct 30 00:04:03.476078 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:03.476044 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-120.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 00:04:03.476078 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.476030 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-120.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 00:04:03.477206 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:03.476032 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-120.ec2.internal.18e325ce07e5f175 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-120.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-120.ec2.internal,},FirstTimestamp:2026-10-30 00:04:03.449590133 +0000 UTC m=+0.392282967,LastTimestamp:2026-10-30 00:04:03.449590133 +0000 UTC m=+0.392282967,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-120.ec2.internal,}" Oct 30 00:04:03.479928 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.479896 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:04:03.482324 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.482290 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:04:03.482396 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.482335 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:04:03.482396 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.482351 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:04:03.482989 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.482971 2580 cpu_manager.go:226] "Starting" policy="none" Oct 30 00:04:03.483056 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.482986 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:04:03.483095 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.482987 2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 00:04:03.483129 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.483107 2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 00:04:03.483667 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.483646 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:04:03.483740 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.483684 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:04:03.483740 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.483700 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:04:03.485373 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.485359 2580 policy_none.go:50] "Start" Oct 30 00:04:03.485452 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.485382 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 00:04:03.485452 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.485392 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 00:04:03.487382 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.487370 2580 policy_none.go:44] "Start" Oct 30 00:04:03.499039 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:03.499013 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 00:04:03.500127 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:03.500102 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-120.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 00:04:03.500198 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:03.500057 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-120.ec2.internal.18e325ce09d933ff default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-120.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-120.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-120.ec2.internal,},FirstTimestamp:2026-10-30 00:04:03.482309631 +0000 UTC m=+0.425002468,LastTimestamp:2026-10-30 00:04:03.482309631 +0000 UTC m=+0.425002468,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-120.ec2.internal,}" Oct 30 00:04:03.500198 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:03.500131 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 00:04:03.501205 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.501190 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sgr7l" Oct 30 00:04:03.515246 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:03.515174 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-120.ec2.internal.18e325ce09d9b6b2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-120.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-120.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-120.ec2.internal,},FirstTimestamp:2026-10-30 00:04:03.48234309 +0000 UTC m=+0.425035924,LastTimestamp:2026-10-30 00:04:03.48234309 +0000 UTC m=+0.425035924,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-120.ec2.internal,}" Oct 30 00:04:03.515707 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.515693 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sgr7l" Oct 30 00:04:03.520726 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.520700 2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 00:04:03.523763 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.523751 2580 manager.go:341] "Starting Device Plugin manager" Oct 30 00:04:03.523836 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:03.523777 2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 00:04:03.523836 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.523785 2580 server.go:87] "Starting device plugin registration server" Oct 30 00:04:03.524013 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.524003 2580 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 00:04:03.524053 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.524015 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 00:04:03.524136 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.524118 2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 00:04:03.524206 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.524186 2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 00:04:03.524206 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.524194 2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 00:04:03.524705 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:03.524687 2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 00:04:03.524769 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:03.524729 2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-120.ec2.internal\" not found" Oct 30 00:04:03.528938 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:03.528864 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-120.ec2.internal.18e325ce09d9ed5a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-120.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-120.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-120.ec2.internal,},FirstTimestamp:2026-10-30 00:04:03.482357082 +0000 UTC m=+0.425049920,LastTimestamp:2026-10-30 00:04:03.482357082 +0000 UTC m=+0.425049920,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-120.ec2.internal,}" Oct 30 00:04:03.589511 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.589473 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 00:04:03.591558 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.590679 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 00:04:03.591558 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.590699 2580 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 00:04:03.591558 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.590780 2580 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 00:04:03.591558 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:03.590822 2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 00:04:03.604876 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.604859 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:04:03.624824 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.624770 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:04:03.631428 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.631407 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:04:03.631510 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.631441 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:04:03.631510 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.631455 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:04:03.631510 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.631484 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-120.ec2.internal" Oct 30 00:04:03.652502 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.652482 2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-120.ec2.internal" Oct 30 00:04:03.652586 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:03.652506 2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-120.ec2.internal\": node \"ip-10-0-143-120.ec2.internal\" not found" Oct 30 00:04:03.690826 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:03.690671 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-120.ec2.internal\" not found" Oct 30 00:04:03.692523 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.692500 2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-120.ec2.internal"] Oct 30 00:04:03.692577 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.692560 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:04:03.693403 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.693391 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:04:03.693449 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.693419 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:04:03.693449 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.693433 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:04:03.695649 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.695638 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:04:03.695806 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.695791 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal" Oct 30 00:04:03.695845 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.695823 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:04:03.696412 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.696390 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:04:03.696412 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.696408 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:04:03.696530 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.696420 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:04:03.696530 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.696428 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:04:03.696530 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.696431 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:04:03.696530 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.696438 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:04:03.698579 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.698562 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-120.ec2.internal" Oct 30 00:04:03.698673 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.698585 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:04:03.699299 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.699272 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:04:03.699377 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.699308 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:04:03.699377 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.699322 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:04:03.724333 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:03.724318 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-120.ec2.internal\" not found" node="ip-10-0-143-120.ec2.internal" Oct 30 00:04:03.727146 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:03.727129 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-120.ec2.internal\" not found" node="ip-10-0-143-120.ec2.internal" Oct 30 00:04:03.763363 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.763339 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/eec55f048e12253ca5b78eb4bbe9a10b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal\" (UID: \"eec55f048e12253ca5b78eb4bbe9a10b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal" Oct 30 00:04:03.763363 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.763365 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eec55f048e12253ca5b78eb4bbe9a10b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal\" (UID: \"eec55f048e12253ca5b78eb4bbe9a10b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal" Oct 30 00:04:03.763534 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.763393 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4dc21da3207f1de3351a28354872ea48-config\") pod \"kube-apiserver-proxy-ip-10-0-143-120.ec2.internal\" (UID: \"4dc21da3207f1de3351a28354872ea48\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-120.ec2.internal" Oct 30 00:04:03.791790 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:03.791765 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-120.ec2.internal\" not found" Oct 30 00:04:03.864378 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.864352 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4dc21da3207f1de3351a28354872ea48-config\") pod \"kube-apiserver-proxy-ip-10-0-143-120.ec2.internal\" (UID: \"4dc21da3207f1de3351a28354872ea48\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-120.ec2.internal" Oct 30 00:04:03.864519 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.864385 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/eec55f048e12253ca5b78eb4bbe9a10b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal\" (UID: \"eec55f048e12253ca5b78eb4bbe9a10b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal" Oct 30 00:04:03.864519 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.864409 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eec55f048e12253ca5b78eb4bbe9a10b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal\" (UID: \"eec55f048e12253ca5b78eb4bbe9a10b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal" Oct 30 00:04:03.864519 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.864491 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4dc21da3207f1de3351a28354872ea48-config\") pod \"kube-apiserver-proxy-ip-10-0-143-120.ec2.internal\" (UID: \"4dc21da3207f1de3351a28354872ea48\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-120.ec2.internal" Oct 30 00:04:03.864630 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.864533 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/eec55f048e12253ca5b78eb4bbe9a10b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal\" (UID: \"eec55f048e12253ca5b78eb4bbe9a10b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal" Oct 30 00:04:03.864630 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:03.864542 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eec55f048e12253ca5b78eb4bbe9a10b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal\" (UID: \"eec55f048e12253ca5b78eb4bbe9a10b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal" Oct 30 00:04:03.892833 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:03.892778 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-120.ec2.internal\" not found" Oct 30 00:04:03.993764 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:03.993734 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-120.ec2.internal\" not found" Oct 30 00:04:04.027961 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:04.027936 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal" Oct 30 00:04:04.031044 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:04.031029 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-120.ec2.internal" Oct 30 00:04:04.094711 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:04.094681 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-120.ec2.internal\" not found" Oct 30 00:04:04.195262 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:04.195202 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-120.ec2.internal\" not found" Oct 30 00:04:04.295896 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:04.295860 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-120.ec2.internal\" not found" Oct 30 00:04:04.387894 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:04.387866 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 00:04:04.388416 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:04.388051 2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 00:04:04.396847 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:04.396827 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-120.ec2.internal\" not found" Oct 30 00:04:04.497366 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:04.497332 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-120.ec2.internal\" not found" Oct 30 00:04:04.521406 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:04.521368 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 23:59:04 +0000 UTC" deadline="2028-05-06 13:14:53.724877056 +0000 UTC" Oct 30 00:04:04.521406 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:04.521400 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13309h10m49.203480409s" Oct 30 00:04:04.572131 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:04.572105 2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:04:04.598125 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:04.598106 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-120.ec2.internal\" not found" Oct 30 00:04:04.608166 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:04.608135 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeec55f048e12253ca5b78eb4bbe9a10b.slice/crio-b1ca01f6bedc6b25cc4de9362a8815a98eafbf74a13540072bca7da1aeefff30 WatchSource:0}: Error finding container b1ca01f6bedc6b25cc4de9362a8815a98eafbf74a13540072bca7da1aeefff30: Status 404 returned error can't find the container with id b1ca01f6bedc6b25cc4de9362a8815a98eafbf74a13540072bca7da1aeefff30 Oct 30 00:04:04.608993 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:04.608970 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dc21da3207f1de3351a28354872ea48.slice/crio-2278a9b540030faf817efed78d6f518955498ae814e860c0c4a05446856e2b59 WatchSource:0}: Error finding container 2278a9b540030faf817efed78d6f518955498ae814e860c0c4a05446856e2b59: Status 404 returned error can't find the container with id 2278a9b540030faf817efed78d6f518955498ae814e860c0c4a05446856e2b59 Oct 30 00:04:04.612429 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:04.612410 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:04:04.625875 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:04.625853 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:04:04.628843 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:04.628815 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:04:04.628940 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:04.628862 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:04:04.628940 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:04.628875 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:04:04.700372 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:04.700333 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-120.ec2.internal\" not found" Oct 30 00:04:04.800596 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:04.800513 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-120.ec2.internal\" not found" Oct 30 00:04:04.900984 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:04.900939 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-120.ec2.internal\" not found" Oct 30 00:04:05.001946 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:05.001902 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-120.ec2.internal\" not found" Oct 30 00:04:05.020852 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.020824 2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:04:05.062082 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.061993 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal" Oct 30 00:04:05.089987 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.089951 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:04:05.091910 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.091890 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-120.ec2.internal" Oct 30 00:04:05.112393 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.112285 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:04:05.253785 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.253752 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:04:05.420324 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.420253 2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:04:05.432147 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.432113 2580 apiserver.go:52] "Watching apiserver" Oct 30 00:04:05.460995 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.460963 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 00:04:05.463248 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.463214 2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 00:04:05.463590 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.463567 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-52qpr","kube-system/konnectivity-agent-q8m2x","openshift-dns/node-resolver-b54v5","openshift-image-registry/node-ca-qs54p","kube-system/global-pull-secret-syncer-zm28x","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f","openshift-cluster-node-tuning-operator/tuned-575b9","openshift-network-operator/iptables-alerter-j6chh","openshift-ovn-kubernetes/ovnkube-node-7k49v","kube-system/kube-apiserver-proxy-ip-10-0-143-120.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal","openshift-multus/multus-56lxt","openshift-network-diagnostics/network-check-target-sxvqx","openshift-multus/multus-additional-cni-plugins-l54xc"] Oct 30 00:04:05.466445 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.466419 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.468819 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.468793 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxvqx" Oct 30 00:04:05.468927 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:05.468884 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxvqx" podUID="a20efc24-e4cb-4075-911e-bfc43dcfc20b" Oct 30 00:04:05.471635 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.471616 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b54v5" Oct 30 00:04:05.474484 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.474387 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qs54p" Oct 30 00:04:05.476675 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.476636 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:05.476783 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:05.476708 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zm28x" podUID="fb8917db-c1ba-43af-995e-1948de69aa29" Oct 30 00:04:05.479164 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.479141 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:05.481585 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.481544 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.483048 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.483001 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:04:05.483316 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.483290 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kct9w\"" Oct 30 00:04:05.483406 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.483366 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 00:04:05.485392 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.485376 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:05.485475 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:05.485429 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-52qpr" podUID="30673bf9-d332-4592-afce-b16db49876cb" Oct 30 00:04:05.488565 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.488241 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q8m2x" Oct 30 00:04:05.491223 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.491200 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:05.494133 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.494111 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j6chh" Oct 30 00:04:05.497022 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.496846 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.524516 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.524495 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 00:04:05.524636 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.524556 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zqmp4" Oct 30 00:04:05.563021 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.562984 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 00:04:05.574825 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.574715 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-run\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.574825 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.574762 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9e7a0c8-43b3-4f8c-8870-505edace7f78-tmp-dir\") pod \"node-resolver-b54v5\" (UID: \"a9e7a0c8-43b3-4f8c-8870-505edace7f78\") " pod="openshift-dns/node-resolver-b54v5" Oct 30 00:04:05.574825 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.574788 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-socket-dir\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:05.575119 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.574859 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/85b08631-ff08-461f-8bda-6a64a609e1f1-serviceca\") pod \"node-ca-qs54p\" (UID: \"85b08631-ff08-461f-8bda-6a64a609e1f1\") " pod="openshift-image-registry/node-ca-qs54p" Oct 30 00:04:05.575119 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.574942 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-sysconfig\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.575119 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.574975 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-var-lib-kubelet\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.575119 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575001 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-run-multus-certs\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.575119 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575024 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2dzh\" (UniqueName: \"kubernetes.io/projected/31c2770e-628c-4bcd-a088-73f190c952cd-kube-api-access-s2dzh\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:05.575119 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575040 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2277c280-7906-4c42-87f1-7941012a2d31-ovnkube-script-lib\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.575119 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575055 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-kubernetes\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.575119 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575078 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpg2j\" (UniqueName: \"kubernetes.io/projected/a20efc24-e4cb-4075-911e-bfc43dcfc20b-kube-api-access-dpg2j\") pod \"network-check-target-sxvqx\" (UID: \"a20efc24-e4cb-4075-911e-bfc43dcfc20b\") " pod="openshift-network-diagnostics/network-check-target-sxvqx" Oct 30 00:04:05.575119 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575094 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0440635e-2cf5-4f40-a4f5-e716376044ba-cni-binary-copy\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:05.575119 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575110 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-var-lib-kubelet\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.575492 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575127 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmw7p\" (UniqueName: \"kubernetes.io/projected/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-kube-api-access-wmw7p\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.575492 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575142 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-sys\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.575492 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575157 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs94l\" (UniqueName: \"kubernetes.io/projected/a9e7a0c8-43b3-4f8c-8870-505edace7f78-kube-api-access-cs94l\") pod \"node-resolver-b54v5\" (UID: \"a9e7a0c8-43b3-4f8c-8870-505edace7f78\") " pod="openshift-dns/node-resolver-b54v5" Oct 30 00:04:05.575492 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575171 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-multus-daemon-config\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.575492 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575193 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-etc-kubernetes\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.575492 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575214 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:05.575492 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575236 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-registration-dir\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:05.575492 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575262 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-var-lib-openvswitch\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.575492 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575299 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2277c280-7906-4c42-87f1-7941012a2d31-env-overrides\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.575492 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575328 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb8917db-c1ba-43af-995e-1948de69aa29-kubelet-config\") pod \"global-pull-secret-syncer-zm28x\" (UID: \"fb8917db-c1ba-43af-995e-1948de69aa29\") " pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:05.575492 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575360 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-multus-socket-dir-parent\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.575492 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575391 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-run-netns\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.575492 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575415 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-multus-conf-dir\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.575492 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575445 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g6m6\" (UniqueName: \"kubernetes.io/projected/85b08631-ff08-461f-8bda-6a64a609e1f1-kube-api-access-4g6m6\") pod \"node-ca-qs54p\" (UID: \"85b08631-ff08-461f-8bda-6a64a609e1f1\") " pod="openshift-image-registry/node-ca-qs54p" Oct 30 00:04:05.575492 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575467 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-etc-selinux\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:05.575492 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575489 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-run-openvswitch\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.576230 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575511 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l48jx\" (UniqueName: \"kubernetes.io/projected/2277c280-7906-4c42-87f1-7941012a2d31-kube-api-access-l48jx\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.576230 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575532 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-sysctl-d\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.576230 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575562 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0440635e-2cf5-4f40-a4f5-e716376044ba-system-cni-dir\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:05.576230 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575584 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-kubelet\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.576230 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575601 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2277c280-7906-4c42-87f1-7941012a2d31-ovnkube-config\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.576230 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575616 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-lib-modules\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.576230 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575633 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-cnibin\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.576230 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575654 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-var-lib-cni-multus\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.576230 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575679 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-node-log\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.576230 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575700 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skw5p\" (UniqueName: \"kubernetes.io/projected/30673bf9-d332-4592-afce-b16db49876cb-kube-api-access-skw5p\") pod \"network-metrics-daemon-52qpr\" (UID: \"30673bf9-d332-4592-afce-b16db49876cb\") " pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:05.576230 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575726 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0440635e-2cf5-4f40-a4f5-e716376044ba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:05.576230 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575750 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-systemd-units\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.576230 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575775 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-log-socket\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.576230 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575800 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-cni-netd\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.576230 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575825 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs\") pod \"network-metrics-daemon-52qpr\" (UID: \"30673bf9-d332-4592-afce-b16db49876cb\") " pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:05.576230 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575860 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-modprobe-d\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.577058 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575885 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fed8e88-bc49-4564-abcc-7d0fff068aa4-tmp\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.577058 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575915 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xgwg\" (UniqueName: \"kubernetes.io/projected/0440635e-2cf5-4f40-a4f5-e716376044ba-kube-api-access-7xgwg\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:05.577058 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575941 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10514d09-1dbf-47c2-bc1b-54a15f384cb3-host-slash\") pod \"iptables-alerter-j6chh\" (UID: \"10514d09-1dbf-47c2-bc1b-54a15f384cb3\") " pod="openshift-network-operator/iptables-alerter-j6chh" Oct 30 00:04:05.577058 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.575967 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-system-cni-dir\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.577058 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.576006 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-cni-bin\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.577058 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.576031 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-host\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.577058 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.576057 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6vsb\" (UniqueName: \"kubernetes.io/projected/10514d09-1dbf-47c2-bc1b-54a15f384cb3-kube-api-access-h6vsb\") pod \"iptables-alerter-j6chh\" (UID: \"10514d09-1dbf-47c2-bc1b-54a15f384cb3\") " pod="openshift-network-operator/iptables-alerter-j6chh" Oct 30 00:04:05.577058 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.576083 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-var-lib-cni-bin\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.577058 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.576107 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85b08631-ff08-461f-8bda-6a64a609e1f1-host\") pod \"node-ca-qs54p\" (UID: \"85b08631-ff08-461f-8bda-6a64a609e1f1\") " pod="openshift-image-registry/node-ca-qs54p" Oct 30 00:04:05.577058 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.576136 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-sys-fs\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:05.577058 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.576163 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb8917db-c1ba-43af-995e-1948de69aa29-original-pull-secret\") pod \"global-pull-secret-syncer-zm28x\" (UID: \"fb8917db-c1ba-43af-995e-1948de69aa29\") " pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:05.577058 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.576189 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-multus-cni-dir\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.577058 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.576214 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-run-k8s-cni-cncf-io\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.577058 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.576239 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-etc-openvswitch\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.577058 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.576266 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5166c4d7-2b82-4bbd-8d58-836cb274ed25-agent-certs\") pod \"konnectivity-agent-q8m2x\" (UID: \"5166c4d7-2b82-4bbd-8d58-836cb274ed25\") " pod="kube-system/konnectivity-agent-q8m2x" Oct 30 00:04:05.577058 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.576291 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-sysctl-conf\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.577058 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.576317 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-tuned\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.577835 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.576351 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxvlt\" (UniqueName: \"kubernetes.io/projected/0fed8e88-bc49-4564-abcc-7d0fff068aa4-kube-api-access-qxvlt\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.577835 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.576377 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb8917db-c1ba-43af-995e-1948de69aa29-dbus\") pod \"global-pull-secret-syncer-zm28x\" (UID: \"fb8917db-c1ba-43af-995e-1948de69aa29\") " pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:05.577835 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.576402 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-cni-binary-copy\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.577835 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.576428 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-slash\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.577835 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.576454 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-run-ovn-kubernetes\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.577835 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.576484 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5166c4d7-2b82-4bbd-8d58-836cb274ed25-konnectivity-ca\") pod \"konnectivity-agent-q8m2x\" (UID: \"5166c4d7-2b82-4bbd-8d58-836cb274ed25\") " pod="kube-system/konnectivity-agent-q8m2x" Oct 30 00:04:05.577835 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.576510 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0440635e-2cf5-4f40-a4f5-e716376044ba-cnibin\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:05.577835 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.576538 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/10514d09-1dbf-47c2-bc1b-54a15f384cb3-iptables-alerter-script\") pod \"iptables-alerter-j6chh\" (UID: \"10514d09-1dbf-47c2-bc1b-54a15f384cb3\") " pod="openshift-network-operator/iptables-alerter-j6chh" Oct 30 00:04:05.577835 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.576571 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-hostroot\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.577835 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.576594 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-run-netns\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.577835 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.576613 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-run-systemd\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.577835 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.576647 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2277c280-7906-4c42-87f1-7941012a2d31-ovn-node-metrics-cert\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.577835 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.576683 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0440635e-2cf5-4f40-a4f5-e716376044ba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:05.577835 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.576711 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-run-ovn\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.577835 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.576737 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.577835 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.576769 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-systemd\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.578692 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.576786 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a9e7a0c8-43b3-4f8c-8870-505edace7f78-hosts-file\") pod \"node-resolver-b54v5\" (UID: \"a9e7a0c8-43b3-4f8c-8870-505edace7f78\") " pod="openshift-dns/node-resolver-b54v5" Oct 30 00:04:05.578692 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.576804 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0440635e-2cf5-4f40-a4f5-e716376044ba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:05.578692 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.576826 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-device-dir\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:05.591887 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.591862 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 00:04:05.592061 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.592044 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 00:04:05.592267 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.592243 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 00:04:05.592378 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.592343 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 00:04:05.592544 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.592527 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 00:04:05.592629 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.592619 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wvd28\"" Oct 30 00:04:05.593114 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.592689 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 00:04:05.593114 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.592716 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 00:04:05.593114 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.592761 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 00:04:05.593114 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.592803 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 00:04:05.593114 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.592809 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 00:04:05.593114 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.592832 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-r8lkn\"" Oct 30 00:04:05.593114 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.592846 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-k8n8m\"" Oct 30 00:04:05.593114 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.592875 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 00:04:05.593114 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.592885 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 00:04:05.593114 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.592894 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:04:05.593114 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.592934 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 00:04:05.593114 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.592873 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 00:04:05.593114 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.592952 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-n7fsr\"" Oct 30 00:04:05.593114 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.592885 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 00:04:05.593114 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.592961 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 00:04:05.593114 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.593031 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 00:04:05.593114 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.593058 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 00:04:05.593114 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.593086 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4xzk4\"" Oct 30 00:04:05.594101 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.593132 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 00:04:05.594101 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.593130 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 00:04:05.594101 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.593177 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 00:04:05.594101 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.593285 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fmnxz\"" Oct 30 00:04:05.594101 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.593339 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 00:04:05.594101 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.593403 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-n6hlg\"" Oct 30 00:04:05.594101 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.593441 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 00:04:05.594101 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.593505 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-h92x8\"" Oct 30 00:04:05.594101 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.593609 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 00:04:05.596455 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.596370 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal" event={"ID":"eec55f048e12253ca5b78eb4bbe9a10b","Type":"ContainerStarted","Data":"b1ca01f6bedc6b25cc4de9362a8815a98eafbf74a13540072bca7da1aeefff30"} Oct 30 00:04:05.597786 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.597758 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-120.ec2.internal" event={"ID":"4dc21da3207f1de3351a28354872ea48","Type":"ContainerStarted","Data":"2278a9b540030faf817efed78d6f518955498ae814e860c0c4a05446856e2b59"} Oct 30 00:04:05.677989 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.677900 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-run\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.677989 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.677960 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9e7a0c8-43b3-4f8c-8870-505edace7f78-tmp-dir\") pod \"node-resolver-b54v5\" (UID: \"a9e7a0c8-43b3-4f8c-8870-505edace7f78\") " pod="openshift-dns/node-resolver-b54v5" Oct 30 00:04:05.678183 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.677991 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-socket-dir\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:05.678183 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678010 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/85b08631-ff08-461f-8bda-6a64a609e1f1-serviceca\") pod \"node-ca-qs54p\" (UID: \"85b08631-ff08-461f-8bda-6a64a609e1f1\") " pod="openshift-image-registry/node-ca-qs54p" Oct 30 00:04:05.678183 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678038 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-sysconfig\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.678183 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678056 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-var-lib-kubelet\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.678183 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678081 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-run-multus-certs\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.678183 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678106 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2dzh\" (UniqueName: \"kubernetes.io/projected/31c2770e-628c-4bcd-a088-73f190c952cd-kube-api-access-s2dzh\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:05.678183 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678130 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2277c280-7906-4c42-87f1-7941012a2d31-ovnkube-script-lib\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.678183 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678145 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-kubernetes\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.678183 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678162 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpg2j\" (UniqueName: \"kubernetes.io/projected/a20efc24-e4cb-4075-911e-bfc43dcfc20b-kube-api-access-dpg2j\") pod \"network-check-target-sxvqx\" (UID: \"a20efc24-e4cb-4075-911e-bfc43dcfc20b\") " pod="openshift-network-diagnostics/network-check-target-sxvqx" Oct 30 00:04:05.678183 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678178 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0440635e-2cf5-4f40-a4f5-e716376044ba-cni-binary-copy\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:05.678565 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678193 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-var-lib-kubelet\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.678565 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678208 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmw7p\" (UniqueName: \"kubernetes.io/projected/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-kube-api-access-wmw7p\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.678565 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678222 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-sys\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.678565 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678247 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs94l\" (UniqueName: \"kubernetes.io/projected/a9e7a0c8-43b3-4f8c-8870-505edace7f78-kube-api-access-cs94l\") pod \"node-resolver-b54v5\" (UID: \"a9e7a0c8-43b3-4f8c-8870-505edace7f78\") " pod="openshift-dns/node-resolver-b54v5" Oct 30 00:04:05.678565 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678273 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-multus-daemon-config\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.678565 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678297 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-etc-kubernetes\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.678565 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678319 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:05.678565 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678337 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-registration-dir\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:05.678565 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678352 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-var-lib-openvswitch\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.678565 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678369 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2277c280-7906-4c42-87f1-7941012a2d31-env-overrides\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.678565 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678392 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb8917db-c1ba-43af-995e-1948de69aa29-kubelet-config\") pod \"global-pull-secret-syncer-zm28x\" (UID: \"fb8917db-c1ba-43af-995e-1948de69aa29\") " pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:05.678565 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678417 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-multus-socket-dir-parent\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.678565 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678438 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-run-netns\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.678565 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678454 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-multus-conf-dir\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.678565 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678472 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g6m6\" (UniqueName: \"kubernetes.io/projected/85b08631-ff08-461f-8bda-6a64a609e1f1-kube-api-access-4g6m6\") pod \"node-ca-qs54p\" (UID: \"85b08631-ff08-461f-8bda-6a64a609e1f1\") " pod="openshift-image-registry/node-ca-qs54p" Oct 30 00:04:05.678565 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678515 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-etc-selinux\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:05.678565 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678532 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-run-openvswitch\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.679180 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678552 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l48jx\" (UniqueName: \"kubernetes.io/projected/2277c280-7906-4c42-87f1-7941012a2d31-kube-api-access-l48jx\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.679180 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678576 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-sysctl-d\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.679180 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678597 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0440635e-2cf5-4f40-a4f5-e716376044ba-system-cni-dir\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:05.679180 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678613 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-kubelet\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.679180 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678628 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2277c280-7906-4c42-87f1-7941012a2d31-ovnkube-config\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.679180 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678644 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-lib-modules\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.679180 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678660 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-cnibin\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.679180 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678674 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-var-lib-cni-multus\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.679180 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678690 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-node-log\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.679180 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678706 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skw5p\" (UniqueName: \"kubernetes.io/projected/30673bf9-d332-4592-afce-b16db49876cb-kube-api-access-skw5p\") pod \"network-metrics-daemon-52qpr\" (UID: \"30673bf9-d332-4592-afce-b16db49876cb\") " pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:05.679180 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678723 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0440635e-2cf5-4f40-a4f5-e716376044ba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:05.679180 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678749 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-systemd-units\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.679180 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678772 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-log-socket\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.679180 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678789 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-cni-netd\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.679180 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678805 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs\") pod \"network-metrics-daemon-52qpr\" (UID: \"30673bf9-d332-4592-afce-b16db49876cb\") " pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:05.679180 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678821 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-modprobe-d\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.679180 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678835 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fed8e88-bc49-4564-abcc-7d0fff068aa4-tmp\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.679771 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678852 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xgwg\" (UniqueName: \"kubernetes.io/projected/0440635e-2cf5-4f40-a4f5-e716376044ba-kube-api-access-7xgwg\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:05.679771 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678869 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10514d09-1dbf-47c2-bc1b-54a15f384cb3-host-slash\") pod \"iptables-alerter-j6chh\" (UID: \"10514d09-1dbf-47c2-bc1b-54a15f384cb3\") " pod="openshift-network-operator/iptables-alerter-j6chh" Oct 30 00:04:05.679771 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678884 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-system-cni-dir\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.679771 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678899 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-cni-bin\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.679771 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678914 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-host\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.679771 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678935 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6vsb\" (UniqueName: \"kubernetes.io/projected/10514d09-1dbf-47c2-bc1b-54a15f384cb3-kube-api-access-h6vsb\") pod \"iptables-alerter-j6chh\" (UID: \"10514d09-1dbf-47c2-bc1b-54a15f384cb3\") " pod="openshift-network-operator/iptables-alerter-j6chh" Oct 30 00:04:05.679771 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678961 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-var-lib-cni-bin\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.679771 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.678981 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85b08631-ff08-461f-8bda-6a64a609e1f1-host\") pod \"node-ca-qs54p\" (UID: \"85b08631-ff08-461f-8bda-6a64a609e1f1\") " pod="openshift-image-registry/node-ca-qs54p" Oct 30 00:04:05.679771 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.679016 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-sys-fs\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:05.679771 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.679044 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb8917db-c1ba-43af-995e-1948de69aa29-original-pull-secret\") pod \"global-pull-secret-syncer-zm28x\" (UID: \"fb8917db-c1ba-43af-995e-1948de69aa29\") " pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:05.679771 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.679076 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-multus-cni-dir\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.679771 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.679093 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-run-k8s-cni-cncf-io\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.679771 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.679110 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-etc-openvswitch\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.679771 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.679132 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5166c4d7-2b82-4bbd-8d58-836cb274ed25-agent-certs\") pod \"konnectivity-agent-q8m2x\" (UID: \"5166c4d7-2b82-4bbd-8d58-836cb274ed25\") " pod="kube-system/konnectivity-agent-q8m2x" Oct 30 00:04:05.679771 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.679158 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-sysctl-conf\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.679771 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.679182 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-tuned\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.679771 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.679208 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxvlt\" (UniqueName: \"kubernetes.io/projected/0fed8e88-bc49-4564-abcc-7d0fff068aa4-kube-api-access-qxvlt\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.680373 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.679224 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb8917db-c1ba-43af-995e-1948de69aa29-dbus\") pod \"global-pull-secret-syncer-zm28x\" (UID: \"fb8917db-c1ba-43af-995e-1948de69aa29\") " pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:05.680373 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.679240 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-cni-binary-copy\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.680373 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.679256 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-slash\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.680373 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.679274 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-run-ovn-kubernetes\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.680373 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.679292 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5166c4d7-2b82-4bbd-8d58-836cb274ed25-konnectivity-ca\") pod \"konnectivity-agent-q8m2x\" (UID: \"5166c4d7-2b82-4bbd-8d58-836cb274ed25\") " pod="kube-system/konnectivity-agent-q8m2x" Oct 30 00:04:05.680373 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.679309 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0440635e-2cf5-4f40-a4f5-e716376044ba-cnibin\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:05.680373 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.679325 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/10514d09-1dbf-47c2-bc1b-54a15f384cb3-iptables-alerter-script\") pod \"iptables-alerter-j6chh\" (UID: \"10514d09-1dbf-47c2-bc1b-54a15f384cb3\") " pod="openshift-network-operator/iptables-alerter-j6chh" Oct 30 00:04:05.680373 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.679357 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-hostroot\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.680373 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.679382 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-run-netns\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.680373 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.679403 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-run-systemd\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.680373 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.679431 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2277c280-7906-4c42-87f1-7941012a2d31-ovn-node-metrics-cert\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.680373 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.679458 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0440635e-2cf5-4f40-a4f5-e716376044ba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:05.680373 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.679478 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-run-ovn\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.680373 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.679495 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.680373 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.679511 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-systemd\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.680373 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.679526 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a9e7a0c8-43b3-4f8c-8870-505edace7f78-hosts-file\") pod \"node-resolver-b54v5\" (UID: \"a9e7a0c8-43b3-4f8c-8870-505edace7f78\") " pod="openshift-dns/node-resolver-b54v5" Oct 30 00:04:05.680373 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.679549 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0440635e-2cf5-4f40-a4f5-e716376044ba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:05.681003 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.679577 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-device-dir\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:05.681003 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.679694 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-device-dir\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:05.681003 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.679822 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-run\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.681003 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.680074 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-systemd-units\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.681003 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.680098 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-var-lib-openvswitch\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.681003 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.680137 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9e7a0c8-43b3-4f8c-8870-505edace7f78-tmp-dir\") pod \"node-resolver-b54v5\" (UID: \"a9e7a0c8-43b3-4f8c-8870-505edace7f78\") " pod="openshift-dns/node-resolver-b54v5" Oct 30 00:04:05.681003 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.680220 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-log-socket\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.681003 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.680264 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-cni-netd\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.681003 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:05.680365 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:04:05.681003 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:05.680438 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs podName:30673bf9-d332-4592-afce-b16db49876cb nodeName:}" failed. No retries permitted until 2026-10-30 00:04:06.180416685 +0000 UTC m=+3.123109525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs") pod "network-metrics-daemon-52qpr" (UID: "30673bf9-d332-4592-afce-b16db49876cb") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:04:05.681003 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.680573 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/85b08631-ff08-461f-8bda-6a64a609e1f1-serviceca\") pod \"node-ca-qs54p\" (UID: \"85b08631-ff08-461f-8bda-6a64a609e1f1\") " pod="openshift-image-registry/node-ca-qs54p" Oct 30 00:04:05.681003 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.680581 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2277c280-7906-4c42-87f1-7941012a2d31-env-overrides\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.681003 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.680641 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-run-multus-certs\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.681003 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.680684 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-socket-dir\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:05.681003 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.680709 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-modprobe-d\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.681003 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.680746 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb8917db-c1ba-43af-995e-1948de69aa29-kubelet-config\") pod \"global-pull-secret-syncer-zm28x\" (UID: \"fb8917db-c1ba-43af-995e-1948de69aa29\") " pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:05.681003 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.680760 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-var-lib-kubelet\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.681825 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.680806 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-multus-socket-dir-parent\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.681825 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.680849 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-run-netns\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.681825 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.680892 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-multus-conf-dir\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.681825 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.681052 2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 00:04:05.681825 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.681093 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-sys\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.681825 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.681164 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-etc-selinux\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:05.681825 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.681244 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-kubernetes\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.681825 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.681276 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2277c280-7906-4c42-87f1-7941012a2d31-ovnkube-script-lib\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.681825 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.681289 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-run-openvswitch\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.681825 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.681385 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-var-lib-kubelet\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.681825 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.681587 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-sysctl-d\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.681825 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.681619 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0440635e-2cf5-4f40-a4f5-e716376044ba-system-cni-dir\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:05.681825 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.681647 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-kubelet\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.681825 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.681714 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb8917db-c1ba-43af-995e-1948de69aa29-dbus\") pod \"global-pull-secret-syncer-zm28x\" (UID: \"fb8917db-c1ba-43af-995e-1948de69aa29\") " pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:05.682539 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.681899 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-multus-daemon-config\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.682539 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.681961 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-etc-kubernetes\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.682539 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.681979 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10514d09-1dbf-47c2-bc1b-54a15f384cb3-host-slash\") pod \"iptables-alerter-j6chh\" (UID: \"10514d09-1dbf-47c2-bc1b-54a15f384cb3\") " pod="openshift-network-operator/iptables-alerter-j6chh" Oct 30 00:04:05.682539 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.682026 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:05.682539 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.682038 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2277c280-7906-4c42-87f1-7941012a2d31-ovnkube-config\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.682539 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.682069 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-system-cni-dir\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.682539 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.682105 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-node-log\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.682539 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.682078 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-run-systemd\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.682539 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.682132 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-cni-bin\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.682539 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.682198 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-hostroot\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.682539 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.682198 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-cnibin\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.682539 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.682224 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0440635e-2cf5-4f40-a4f5-e716376044ba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:05.682539 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.682232 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-var-lib-cni-multus\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.682539 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.682253 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-run-netns\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.682539 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.682310 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-systemd\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.682539 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.682394 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-var-lib-cni-bin\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.682539 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.682510 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0440635e-2cf5-4f40-a4f5-e716376044ba-cnibin\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:05.683405 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.682623 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0440635e-2cf5-4f40-a4f5-e716376044ba-cni-binary-copy\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:05.683405 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.682686 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-run-ovn-kubernetes\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.683405 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.682764 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-cni-binary-copy\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.683405 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.682795 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-lib-modules\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.683405 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.682816 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5166c4d7-2b82-4bbd-8d58-836cb274ed25-konnectivity-ca\") pod \"konnectivity-agent-q8m2x\" (UID: \"5166c4d7-2b82-4bbd-8d58-836cb274ed25\") " pod="kube-system/konnectivity-agent-q8m2x" Oct 30 00:04:05.683405 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.682841 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-host\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.683405 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.682885 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.683405 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.682909 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-etc-openvswitch\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.683405 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.682942 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-sysconfig\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.683405 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.682949 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0440635e-2cf5-4f40-a4f5-e716376044ba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:05.683405 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.682950 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85b08631-ff08-461f-8bda-6a64a609e1f1-host\") pod \"node-ca-qs54p\" (UID: \"85b08631-ff08-461f-8bda-6a64a609e1f1\") " pod="openshift-image-registry/node-ca-qs54p" Oct 30 00:04:05.683405 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.683082 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-multus-cni-dir\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.683405 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.683154 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-sysctl-conf\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.683405 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:05.683171 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:04:05.683405 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.683172 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/10514d09-1dbf-47c2-bc1b-54a15f384cb3-iptables-alerter-script\") pod \"iptables-alerter-j6chh\" (UID: \"10514d09-1dbf-47c2-bc1b-54a15f384cb3\") " pod="openshift-network-operator/iptables-alerter-j6chh" Oct 30 00:04:05.683405 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.682073 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-registration-dir\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:05.683405 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.683285 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a9e7a0c8-43b3-4f8c-8870-505edace7f78-hosts-file\") pod \"node-resolver-b54v5\" (UID: \"a9e7a0c8-43b3-4f8c-8870-505edace7f78\") " pod="openshift-dns/node-resolver-b54v5" Oct 30 00:04:05.684073 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:05.683279 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb8917db-c1ba-43af-995e-1948de69aa29-original-pull-secret podName:fb8917db-c1ba-43af-995e-1948de69aa29 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:06.183262321 +0000 UTC m=+3.125955158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb8917db-c1ba-43af-995e-1948de69aa29-original-pull-secret") pod "global-pull-secret-syncer-zm28x" (UID: "fb8917db-c1ba-43af-995e-1948de69aa29") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:04:05.684073 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.683345 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-sys-fs\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:05.684073 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.683375 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-run-ovn\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.684073 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.683385 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-slash\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.684073 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.683425 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0440635e-2cf5-4f40-a4f5-e716376044ba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:05.684073 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.683452 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-run-k8s-cni-cncf-io\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.685700 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.685660 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fed8e88-bc49-4564-abcc-7d0fff068aa4-tmp\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.686549 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.686526 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-tuned\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.686705 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.686631 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2277c280-7906-4c42-87f1-7941012a2d31-ovn-node-metrics-cert\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.688041 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.688020 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5166c4d7-2b82-4bbd-8d58-836cb274ed25-agent-certs\") pod \"konnectivity-agent-q8m2x\" (UID: \"5166c4d7-2b82-4bbd-8d58-836cb274ed25\") " pod="kube-system/konnectivity-agent-q8m2x" Oct 30 00:04:05.704883 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:05.704861 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:04:05.705005 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:05.704891 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:04:05.705005 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:05.704905 2580 projected.go:196] Error preparing data for projected volume kube-api-access-dpg2j for pod openshift-network-diagnostics/network-check-target-sxvqx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:04:05.705005 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:05.704974 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a20efc24-e4cb-4075-911e-bfc43dcfc20b-kube-api-access-dpg2j podName:a20efc24-e4cb-4075-911e-bfc43dcfc20b nodeName:}" failed. No retries permitted until 2026-10-30 00:04:06.204957259 +0000 UTC m=+3.147650099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dpg2j" (UniqueName: "kubernetes.io/projected/a20efc24-e4cb-4075-911e-bfc43dcfc20b-kube-api-access-dpg2j") pod "network-check-target-sxvqx" (UID: "a20efc24-e4cb-4075-911e-bfc43dcfc20b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:04:05.706522 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.706493 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g6m6\" (UniqueName: \"kubernetes.io/projected/85b08631-ff08-461f-8bda-6a64a609e1f1-kube-api-access-4g6m6\") pod \"node-ca-qs54p\" (UID: \"85b08631-ff08-461f-8bda-6a64a609e1f1\") " pod="openshift-image-registry/node-ca-qs54p" Oct 30 00:04:05.706652 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.706547 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2dzh\" (UniqueName: \"kubernetes.io/projected/31c2770e-628c-4bcd-a088-73f190c952cd-kube-api-access-s2dzh\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:05.708826 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.708800 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmw7p\" (UniqueName: \"kubernetes.io/projected/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-kube-api-access-wmw7p\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.709356 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.709298 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6vsb\" (UniqueName: \"kubernetes.io/projected/10514d09-1dbf-47c2-bc1b-54a15f384cb3-kube-api-access-h6vsb\") pod \"iptables-alerter-j6chh\" (UID: \"10514d09-1dbf-47c2-bc1b-54a15f384cb3\") " pod="openshift-network-operator/iptables-alerter-j6chh" Oct 30 00:04:05.710013 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.709988 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xgwg\" (UniqueName: \"kubernetes.io/projected/0440635e-2cf5-4f40-a4f5-e716376044ba-kube-api-access-7xgwg\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:05.710108 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.710048 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skw5p\" (UniqueName: \"kubernetes.io/projected/30673bf9-d332-4592-afce-b16db49876cb-kube-api-access-skw5p\") pod \"network-metrics-daemon-52qpr\" (UID: \"30673bf9-d332-4592-afce-b16db49876cb\") " pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:05.710320 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.710270 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l48jx\" (UniqueName: \"kubernetes.io/projected/2277c280-7906-4c42-87f1-7941012a2d31-kube-api-access-l48jx\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.711374 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.711354 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs94l\" (UniqueName: \"kubernetes.io/projected/a9e7a0c8-43b3-4f8c-8870-505edace7f78-kube-api-access-cs94l\") pod \"node-resolver-b54v5\" (UID: \"a9e7a0c8-43b3-4f8c-8870-505edace7f78\") " pod="openshift-dns/node-resolver-b54v5" Oct 30 00:04:05.711442 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.711419 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxvlt\" (UniqueName: \"kubernetes.io/projected/0fed8e88-bc49-4564-abcc-7d0fff068aa4-kube-api-access-qxvlt\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.780761 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.780718 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:05.791560 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.791532 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b54v5" Oct 30 00:04:05.792874 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:05.792841 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fed8e88_bc49_4564_abcc_7d0fff068aa4.slice/crio-e284b2defae875dd7e7df27643e9f3b13a295ccf233b443acc08560ed454e6c6 WatchSource:0}: Error finding container e284b2defae875dd7e7df27643e9f3b13a295ccf233b443acc08560ed454e6c6: Status 404 returned error can't find the container with id e284b2defae875dd7e7df27643e9f3b13a295ccf233b443acc08560ed454e6c6 Oct 30 00:04:05.800037 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.800014 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qs54p" Oct 30 00:04:05.803245 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:05.803217 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e7a0c8_43b3_4f8c_8870_505edace7f78.slice/crio-43abae8136f399bc01fc7520685f1890ebb0d48a0060151959535418cf4d3f98 WatchSource:0}: Error finding container 43abae8136f399bc01fc7520685f1890ebb0d48a0060151959535418cf4d3f98: Status 404 returned error can't find the container with id 43abae8136f399bc01fc7520685f1890ebb0d48a0060151959535418cf4d3f98 Oct 30 00:04:05.810573 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:05.810550 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b08631_ff08_461f_8bda_6a64a609e1f1.slice/crio-65987899f63eeb90b91417e14c7c9a09d3a46338421ca761818ad060f4639501 WatchSource:0}: Error finding container 65987899f63eeb90b91417e14c7c9a09d3a46338421ca761818ad060f4639501: Status 404 returned error can't find the container with id 65987899f63eeb90b91417e14c7c9a09d3a46338421ca761818ad060f4639501 Oct 30 00:04:05.812341 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.812319 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:05.823426 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:05.823402 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31c2770e_628c_4bcd_a088_73f190c952cd.slice/crio-5306018f5ceb7f93fc5e1cf1adc00fb3c87e0f81f729043a5983f1feb3fd0757 WatchSource:0}: Error finding container 5306018f5ceb7f93fc5e1cf1adc00fb3c87e0f81f729043a5983f1feb3fd0757: Status 404 returned error can't find the container with id 5306018f5ceb7f93fc5e1cf1adc00fb3c87e0f81f729043a5983f1feb3fd0757 Oct 30 00:04:05.837114 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.837091 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q8m2x" Oct 30 00:04:05.845834 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.845811 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:05.851443 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.851417 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j6chh" Oct 30 00:04:05.854126 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.854107 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-56lxt" Oct 30 00:04:05.855914 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:05.855888 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5166c4d7_2b82_4bbd_8d58_836cb274ed25.slice/crio-baa7cbab439b2099259e9678bc017a1fb76d1b2165ad948b41c6994d2b387a38 WatchSource:0}: Error finding container baa7cbab439b2099259e9678bc017a1fb76d1b2165ad948b41c6994d2b387a38: Status 404 returned error can't find the container with id baa7cbab439b2099259e9678bc017a1fb76d1b2165ad948b41c6994d2b387a38 Oct 30 00:04:05.859073 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:05.858949 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2277c280_7906_4c42_87f1_7941012a2d31.slice/crio-ba818f2ebc5b2cbeb9424de1c06d70910d2c2f9cc6094887b0ca46cef7294654 WatchSource:0}: Error finding container ba818f2ebc5b2cbeb9424de1c06d70910d2c2f9cc6094887b0ca46cef7294654: Status 404 returned error can't find the container with id ba818f2ebc5b2cbeb9424de1c06d70910d2c2f9cc6094887b0ca46cef7294654 Oct 30 00:04:05.863336 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:05.863318 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:05.866816 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:05.866792 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27614fa1_66da_4d21_a0d1_0ac85a9bdd13.slice/crio-709f1626e391eae94355f8e5c2b187186a5c38c08846b951d629e2f53e8fa9c1 WatchSource:0}: Error finding container 709f1626e391eae94355f8e5c2b187186a5c38c08846b951d629e2f53e8fa9c1: Status 404 returned error can't find the container with id 709f1626e391eae94355f8e5c2b187186a5c38c08846b951d629e2f53e8fa9c1 Oct 30 00:04:05.868178 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:05.868110 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10514d09_1dbf_47c2_bc1b_54a15f384cb3.slice/crio-4d032dee89180d1a063f800c46cf9bee3a78b9a8137597408ad3e4e26624ebaf WatchSource:0}: Error finding container 4d032dee89180d1a063f800c46cf9bee3a78b9a8137597408ad3e4e26624ebaf: Status 404 returned error can't find the container with id 4d032dee89180d1a063f800c46cf9bee3a78b9a8137597408ad3e4e26624ebaf Oct 30 00:04:05.875499 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:05.875359 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0440635e_2cf5_4f40_a4f5_e716376044ba.slice/crio-0962758c857bfd33f7077d1299e3caff3ae85f46d85bfa240cc0362d3a74fe0c WatchSource:0}: Error finding container 0962758c857bfd33f7077d1299e3caff3ae85f46d85bfa240cc0362d3a74fe0c: Status 404 returned error can't find the container with id 0962758c857bfd33f7077d1299e3caff3ae85f46d85bfa240cc0362d3a74fe0c Oct 30 00:04:06.182945 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:06.182919 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs\") pod \"network-metrics-daemon-52qpr\" (UID: \"30673bf9-d332-4592-afce-b16db49876cb\") " pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:06.183122 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:06.183106 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:04:06.183185 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:06.183174 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs podName:30673bf9-d332-4592-afce-b16db49876cb nodeName:}" failed. No retries permitted until 2026-10-30 00:04:07.183156457 +0000 UTC m=+4.125849292 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs") pod "network-metrics-daemon-52qpr" (UID: "30673bf9-d332-4592-afce-b16db49876cb") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:04:06.286819 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:06.285891 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpg2j\" (UniqueName: \"kubernetes.io/projected/a20efc24-e4cb-4075-911e-bfc43dcfc20b-kube-api-access-dpg2j\") pod \"network-check-target-sxvqx\" (UID: \"a20efc24-e4cb-4075-911e-bfc43dcfc20b\") " pod="openshift-network-diagnostics/network-check-target-sxvqx" Oct 30 00:04:06.286819 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:06.286038 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb8917db-c1ba-43af-995e-1948de69aa29-original-pull-secret\") pod \"global-pull-secret-syncer-zm28x\" (UID: \"fb8917db-c1ba-43af-995e-1948de69aa29\") " pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:06.286819 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:06.286262 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:04:06.286819 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:06.286355 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb8917db-c1ba-43af-995e-1948de69aa29-original-pull-secret podName:fb8917db-c1ba-43af-995e-1948de69aa29 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:07.286315818 +0000 UTC m=+4.229008663 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb8917db-c1ba-43af-995e-1948de69aa29-original-pull-secret") pod "global-pull-secret-syncer-zm28x" (UID: "fb8917db-c1ba-43af-995e-1948de69aa29") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:04:06.286819 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:06.286722 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:04:06.286819 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:06.286739 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:04:06.286819 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:06.286750 2580 projected.go:196] Error preparing data for projected volume kube-api-access-dpg2j for pod openshift-network-diagnostics/network-check-target-sxvqx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:04:06.286819 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:06.286787 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a20efc24-e4cb-4075-911e-bfc43dcfc20b-kube-api-access-dpg2j podName:a20efc24-e4cb-4075-911e-bfc43dcfc20b nodeName:}" failed. No retries permitted until 2026-10-30 00:04:07.286775434 +0000 UTC m=+4.229468258 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dpg2j" (UniqueName: "kubernetes.io/projected/a20efc24-e4cb-4075-911e-bfc43dcfc20b-kube-api-access-dpg2j") pod "network-check-target-sxvqx" (UID: "a20efc24-e4cb-4075-911e-bfc43dcfc20b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:04:06.529436 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:06.529089 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 23:59:06 +0000 UTC" deadline="2028-07-04 01:53:08.259417053 +0000 UTC" Oct 30 00:04:06.529436 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:06.529383 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14713h49m1.730039205s" Oct 30 00:04:06.604924 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:06.604083 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-120.ec2.internal" event={"ID":"4dc21da3207f1de3351a28354872ea48","Type":"ContainerStarted","Data":"72eb34cdcfe154e33becddd1ee0821fb2adfbbd8519818fdf2e2557a8d985f4c"} Oct 30 00:04:06.615994 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:06.615905 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-56lxt" event={"ID":"27614fa1-66da-4d21-a0d1-0ac85a9bdd13","Type":"ContainerStarted","Data":"709f1626e391eae94355f8e5c2b187186a5c38c08846b951d629e2f53e8fa9c1"} Oct 30 00:04:06.621806 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:06.620799 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j6chh" event={"ID":"10514d09-1dbf-47c2-bc1b-54a15f384cb3","Type":"ContainerStarted","Data":"4d032dee89180d1a063f800c46cf9bee3a78b9a8137597408ad3e4e26624ebaf"} Oct 30 00:04:06.630263 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:06.630132 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" event={"ID":"2277c280-7906-4c42-87f1-7941012a2d31","Type":"ContainerStarted","Data":"ba818f2ebc5b2cbeb9424de1c06d70910d2c2f9cc6094887b0ca46cef7294654"} Oct 30 00:04:06.635554 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:06.635454 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" event={"ID":"31c2770e-628c-4bcd-a088-73f190c952cd","Type":"ContainerStarted","Data":"5306018f5ceb7f93fc5e1cf1adc00fb3c87e0f81f729043a5983f1feb3fd0757"} Oct 30 00:04:06.646522 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:06.646244 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-575b9" event={"ID":"0fed8e88-bc49-4564-abcc-7d0fff068aa4","Type":"ContainerStarted","Data":"e284b2defae875dd7e7df27643e9f3b13a295ccf233b443acc08560ed454e6c6"} Oct 30 00:04:06.647261 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:06.646961 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-120.ec2.internal" podStartSLOduration=1.646922419 podStartE2EDuration="1.646922419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:04:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:06.64513845 +0000 UTC m=+3.587831297" watchObservedRunningTime="2026-10-30 00:04:06.646922419 +0000 UTC m=+3.589615268" Oct 30 00:04:06.650676 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:06.650592 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l54xc" event={"ID":"0440635e-2cf5-4f40-a4f5-e716376044ba","Type":"ContainerStarted","Data":"0962758c857bfd33f7077d1299e3caff3ae85f46d85bfa240cc0362d3a74fe0c"} Oct 30 00:04:06.654732 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:06.654680 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q8m2x" event={"ID":"5166c4d7-2b82-4bbd-8d58-836cb274ed25","Type":"ContainerStarted","Data":"baa7cbab439b2099259e9678bc017a1fb76d1b2165ad948b41c6994d2b387a38"} Oct 30 00:04:06.656982 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:06.656900 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qs54p" event={"ID":"85b08631-ff08-461f-8bda-6a64a609e1f1","Type":"ContainerStarted","Data":"65987899f63eeb90b91417e14c7c9a09d3a46338421ca761818ad060f4639501"} Oct 30 00:04:06.658826 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:06.658763 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b54v5" event={"ID":"a9e7a0c8-43b3-4f8c-8870-505edace7f78","Type":"ContainerStarted","Data":"43abae8136f399bc01fc7520685f1890ebb0d48a0060151959535418cf4d3f98"} Oct 30 00:04:07.194158 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:07.193493 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs\") pod \"network-metrics-daemon-52qpr\" (UID: \"30673bf9-d332-4592-afce-b16db49876cb\") " pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:07.194158 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:07.193710 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:04:07.194158 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:07.193787 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs podName:30673bf9-d332-4592-afce-b16db49876cb nodeName:}" failed. No retries permitted until 2026-10-30 00:04:09.193758567 +0000 UTC m=+6.136451411 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs") pod "network-metrics-daemon-52qpr" (UID: "30673bf9-d332-4592-afce-b16db49876cb") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:04:07.294641 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:07.294598 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpg2j\" (UniqueName: \"kubernetes.io/projected/a20efc24-e4cb-4075-911e-bfc43dcfc20b-kube-api-access-dpg2j\") pod \"network-check-target-sxvqx\" (UID: \"a20efc24-e4cb-4075-911e-bfc43dcfc20b\") " pod="openshift-network-diagnostics/network-check-target-sxvqx" Oct 30 00:04:07.294821 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:07.294683 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb8917db-c1ba-43af-995e-1948de69aa29-original-pull-secret\") pod \"global-pull-secret-syncer-zm28x\" (UID: \"fb8917db-c1ba-43af-995e-1948de69aa29\") " pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:07.294821 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:07.294794 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:04:07.294896 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:07.294843 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb8917db-c1ba-43af-995e-1948de69aa29-original-pull-secret podName:fb8917db-c1ba-43af-995e-1948de69aa29 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:09.294830484 +0000 UTC m=+6.237523321 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb8917db-c1ba-43af-995e-1948de69aa29-original-pull-secret") pod "global-pull-secret-syncer-zm28x" (UID: "fb8917db-c1ba-43af-995e-1948de69aa29") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:04:07.301407 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:07.301206 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:04:07.301407 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:07.301248 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:04:07.301407 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:07.301263 2580 projected.go:196] Error preparing data for projected volume kube-api-access-dpg2j for pod openshift-network-diagnostics/network-check-target-sxvqx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:04:07.301407 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:07.301363 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a20efc24-e4cb-4075-911e-bfc43dcfc20b-kube-api-access-dpg2j podName:a20efc24-e4cb-4075-911e-bfc43dcfc20b nodeName:}" failed. No retries permitted until 2026-10-30 00:04:09.301333772 +0000 UTC m=+6.244026620 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dpg2j" (UniqueName: "kubernetes.io/projected/a20efc24-e4cb-4075-911e-bfc43dcfc20b-kube-api-access-dpg2j") pod "network-check-target-sxvqx" (UID: "a20efc24-e4cb-4075-911e-bfc43dcfc20b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:04:07.530239 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:07.530105 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 23:59:06 +0000 UTC" deadline="2028-06-22 11:28:10.357581654 +0000 UTC" Oct 30 00:04:07.530239 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:07.530167 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14435h24m2.827419063s" Oct 30 00:04:07.594596 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:07.593497 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:07.594596 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:07.593653 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-52qpr" podUID="30673bf9-d332-4592-afce-b16db49876cb" Oct 30 00:04:07.594596 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:07.594095 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxvqx" Oct 30 00:04:07.594596 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:07.594195 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxvqx" podUID="a20efc24-e4cb-4075-911e-bfc43dcfc20b" Oct 30 00:04:07.594596 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:07.594300 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:07.594596 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:07.594444 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zm28x" podUID="fb8917db-c1ba-43af-995e-1948de69aa29" Oct 30 00:04:07.669020 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:07.668328 2580 generic.go:356] "Generic (PLEG): container finished" podID="eec55f048e12253ca5b78eb4bbe9a10b" containerID="7e94b648bc04fb39748d4f157f7d37462ff159ab123a148d6764252d93c7f4c4" exitCode=0 Oct 30 00:04:07.669497 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:07.669247 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal" event={"ID":"eec55f048e12253ca5b78eb4bbe9a10b","Type":"ContainerDied","Data":"7e94b648bc04fb39748d4f157f7d37462ff159ab123a148d6764252d93c7f4c4"} Oct 30 00:04:08.690940 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:08.690901 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal" event={"ID":"eec55f048e12253ca5b78eb4bbe9a10b","Type":"ContainerStarted","Data":"73d4eb4fead93bc8647a781334f5b6ca16e62c7b95f4b1d47a32f764c41ae8ec"} Oct 30 00:04:09.210865 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:09.210764 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs\") pod \"network-metrics-daemon-52qpr\" (UID: \"30673bf9-d332-4592-afce-b16db49876cb\") " pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:09.211061 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:09.210972 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:04:09.211061 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:09.211043 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs podName:30673bf9-d332-4592-afce-b16db49876cb nodeName:}" failed. No retries permitted until 2026-10-30 00:04:13.21102274 +0000 UTC m=+10.153715578 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs") pod "network-metrics-daemon-52qpr" (UID: "30673bf9-d332-4592-afce-b16db49876cb") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:04:09.311467 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:09.311266 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb8917db-c1ba-43af-995e-1948de69aa29-original-pull-secret\") pod \"global-pull-secret-syncer-zm28x\" (UID: \"fb8917db-c1ba-43af-995e-1948de69aa29\") " pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:09.311467 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:09.311348 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpg2j\" (UniqueName: \"kubernetes.io/projected/a20efc24-e4cb-4075-911e-bfc43dcfc20b-kube-api-access-dpg2j\") pod \"network-check-target-sxvqx\" (UID: \"a20efc24-e4cb-4075-911e-bfc43dcfc20b\") " pod="openshift-network-diagnostics/network-check-target-sxvqx" Oct 30 00:04:09.311702 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:09.311539 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:04:09.311702 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:09.311559 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:04:09.311702 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:09.311573 2580 projected.go:196] Error preparing data for projected volume kube-api-access-dpg2j for pod openshift-network-diagnostics/network-check-target-sxvqx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:04:09.311702 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:09.311637 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a20efc24-e4cb-4075-911e-bfc43dcfc20b-kube-api-access-dpg2j podName:a20efc24-e4cb-4075-911e-bfc43dcfc20b nodeName:}" failed. No retries permitted until 2026-10-30 00:04:13.311615636 +0000 UTC m=+10.254308482 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dpg2j" (UniqueName: "kubernetes.io/projected/a20efc24-e4cb-4075-911e-bfc43dcfc20b-kube-api-access-dpg2j") pod "network-check-target-sxvqx" (UID: "a20efc24-e4cb-4075-911e-bfc43dcfc20b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:04:09.312105 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:09.312085 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:04:09.312174 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:09.312142 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb8917db-c1ba-43af-995e-1948de69aa29-original-pull-secret podName:fb8917db-c1ba-43af-995e-1948de69aa29 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:13.312127605 +0000 UTC m=+10.254820441 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb8917db-c1ba-43af-995e-1948de69aa29-original-pull-secret") pod "global-pull-secret-syncer-zm28x" (UID: "fb8917db-c1ba-43af-995e-1948de69aa29") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:04:09.594598 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:09.594520 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:09.594761 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:09.594695 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-52qpr" podUID="30673bf9-d332-4592-afce-b16db49876cb" Oct 30 00:04:09.595202 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:09.595175 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxvqx" Oct 30 00:04:09.595312 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:09.595292 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxvqx" podUID="a20efc24-e4cb-4075-911e-bfc43dcfc20b" Oct 30 00:04:09.595379 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:09.595347 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:09.595429 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:09.595418 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zm28x" podUID="fb8917db-c1ba-43af-995e-1948de69aa29" Oct 30 00:04:09.696230 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:09.696187 2580 generic.go:356] "Generic (PLEG): container finished" podID="0440635e-2cf5-4f40-a4f5-e716376044ba" containerID="ac0536e875bde4894f14e79d5bff278d8724934dd74bd6288908f15c09fb34e0" exitCode=0 Oct 30 00:04:09.697358 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:09.697311 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l54xc" event={"ID":"0440635e-2cf5-4f40-a4f5-e716376044ba","Type":"ContainerDied","Data":"ac0536e875bde4894f14e79d5bff278d8724934dd74bd6288908f15c09fb34e0"} Oct 30 00:04:09.769099 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:09.769038 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal" podStartSLOduration=4.769015288 podStartE2EDuration="4.769015288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:04:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:08.732840384 +0000 UTC m=+5.675533233" watchObservedRunningTime="2026-10-30 00:04:09.769015288 +0000 UTC m=+6.711708135" Oct 30 00:04:10.703887 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:10.703251 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q8m2x" event={"ID":"5166c4d7-2b82-4bbd-8d58-836cb274ed25","Type":"ContainerStarted","Data":"44ef3b5478343a7800dd596e0410cf9aa87aee53a3cc31099394adde9b257eb0"} Oct 30 00:04:10.745175 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:10.744584 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-q8m2x" podStartSLOduration=2.638953045 podStartE2EDuration="6.744566448s" totalImagesPullingTime="4.105613403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:04:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:10.744146675 +0000 UTC m=+7.686839522" watchObservedRunningTime="2026-10-30 00:04:10.744566448 +0000 UTC m=+7.687259294" Oct 30 00:04:11.593827 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:11.593786 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:11.594039 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:11.593925 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zm28x" podUID="fb8917db-c1ba-43af-995e-1948de69aa29" Oct 30 00:04:11.594359 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:11.594338 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:11.594479 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:11.594459 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-52qpr" podUID="30673bf9-d332-4592-afce-b16db49876cb" Oct 30 00:04:11.594539 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:11.594524 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxvqx" Oct 30 00:04:11.594609 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:11.594592 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxvqx" podUID="a20efc24-e4cb-4075-911e-bfc43dcfc20b" Oct 30 00:04:11.710012 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:11.709974 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qs54p" event={"ID":"85b08631-ff08-461f-8bda-6a64a609e1f1","Type":"ContainerStarted","Data":"164f893e92aef7451f4f12acaa41d8b672562b7336e224404540a11eb501be8a"} Oct 30 00:04:12.717584 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:12.717548 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" event={"ID":"31c2770e-628c-4bcd-a088-73f190c952cd","Type":"ContainerStarted","Data":"eec1a75729c4a84d78acc6cb7d906eb46266ed16b60968e33fa549ae70f8d575"} Oct 30 00:04:13.252250 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:13.252213 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs\") pod \"network-metrics-daemon-52qpr\" (UID: \"30673bf9-d332-4592-afce-b16db49876cb\") " pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:13.252425 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:13.252391 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:04:13.252499 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:13.252457 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs podName:30673bf9-d332-4592-afce-b16db49876cb nodeName:}" failed. No retries permitted until 2026-10-30 00:04:21.252437244 +0000 UTC m=+18.195130090 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs") pod "network-metrics-daemon-52qpr" (UID: "30673bf9-d332-4592-afce-b16db49876cb") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:04:13.353986 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:13.353083 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb8917db-c1ba-43af-995e-1948de69aa29-original-pull-secret\") pod \"global-pull-secret-syncer-zm28x\" (UID: \"fb8917db-c1ba-43af-995e-1948de69aa29\") " pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:13.353986 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:13.353173 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpg2j\" (UniqueName: \"kubernetes.io/projected/a20efc24-e4cb-4075-911e-bfc43dcfc20b-kube-api-access-dpg2j\") pod \"network-check-target-sxvqx\" (UID: \"a20efc24-e4cb-4075-911e-bfc43dcfc20b\") " pod="openshift-network-diagnostics/network-check-target-sxvqx" Oct 30 00:04:13.353986 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:13.353367 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:04:13.353986 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:13.353388 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:04:13.353986 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:13.353402 2580 projected.go:196] Error preparing data for projected volume kube-api-access-dpg2j for pod openshift-network-diagnostics/network-check-target-sxvqx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:04:13.353986 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:13.353464 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a20efc24-e4cb-4075-911e-bfc43dcfc20b-kube-api-access-dpg2j podName:a20efc24-e4cb-4075-911e-bfc43dcfc20b nodeName:}" failed. No retries permitted until 2026-10-30 00:04:21.353442273 +0000 UTC m=+18.296135110 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dpg2j" (UniqueName: "kubernetes.io/projected/a20efc24-e4cb-4075-911e-bfc43dcfc20b-kube-api-access-dpg2j") pod "network-check-target-sxvqx" (UID: "a20efc24-e4cb-4075-911e-bfc43dcfc20b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:04:13.353986 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:13.353889 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:04:13.353986 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:13.353945 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb8917db-c1ba-43af-995e-1948de69aa29-original-pull-secret podName:fb8917db-c1ba-43af-995e-1948de69aa29 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:21.353929022 +0000 UTC m=+18.296621845 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb8917db-c1ba-43af-995e-1948de69aa29-original-pull-secret") pod "global-pull-secret-syncer-zm28x" (UID: "fb8917db-c1ba-43af-995e-1948de69aa29") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:04:13.360761 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:13.360028 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-q8m2x" Oct 30 00:04:13.361292 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:13.361125 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-q8m2x" Oct 30 00:04:13.402481 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:13.401233 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qs54p" podStartSLOduration=4.432300467 podStartE2EDuration="9.401211055s" totalImagesPullingTime="4.968910588s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:04:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:11.776124996 +0000 UTC m=+8.718817845" watchObservedRunningTime="2026-10-30 00:04:13.401211055 +0000 UTC m=+10.343903904" Oct 30 00:04:13.592202 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:13.592064 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:13.592202 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:13.592064 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxvqx" Oct 30 00:04:13.593477 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:13.593265 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zm28x" podUID="fb8917db-c1ba-43af-995e-1948de69aa29" Oct 30 00:04:13.593477 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:13.593295 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxvqx" podUID="a20efc24-e4cb-4075-911e-bfc43dcfc20b" Oct 30 00:04:13.593477 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:13.593347 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:13.593477 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:13.593434 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-52qpr" podUID="30673bf9-d332-4592-afce-b16db49876cb" Oct 30 00:04:13.935719 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:13.935642 2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 00:04:14.530244 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:14.529992 2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T00:04:13.935661031Z","UUID":"9097c550-2a9f-495d-8245-3c5f4bed393d","Handler":null,"Name":"","Endpoint":""} Oct 30 00:04:14.531592 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:14.531570 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 00:04:14.531714 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:14.531600 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 00:04:14.724212 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:14.724176 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-56lxt" event={"ID":"27614fa1-66da-4d21-a0d1-0ac85a9bdd13","Type":"ContainerStarted","Data":"cf730fa6b64a45ef7f745d38588f191ed57efa3ed1d0ab601edafe20e3c2d438"} Oct 30 00:04:14.727307 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:14.727278 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" event={"ID":"31c2770e-628c-4bcd-a088-73f190c952cd","Type":"ContainerStarted","Data":"e0b87c083bd5b0b23fe84f93bae1c347d04543db635af8ea22b0e974628ab704"} Oct 30 00:04:14.729715 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:14.729688 2580 generic.go:356] "Generic (PLEG): container finished" podID="0440635e-2cf5-4f40-a4f5-e716376044ba" containerID="ce2133e430132ed8fc264df5d195ade3cd63bbcf9c3579f1aeb3480573c9f371" exitCode=0 Oct 30 00:04:14.729800 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:14.729763 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l54xc" event={"ID":"0440635e-2cf5-4f40-a4f5-e716376044ba","Type":"ContainerDied","Data":"ce2133e430132ed8fc264df5d195ade3cd63bbcf9c3579f1aeb3480573c9f371"} Oct 30 00:04:14.732546 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:14.732524 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b54v5" event={"ID":"a9e7a0c8-43b3-4f8c-8870-505edace7f78","Type":"ContainerStarted","Data":"6c95151a5234bf926e7c7b8bcbb92fbbcae378e23bd30472a0876a1c7106ae42"} Oct 30 00:04:14.802227 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:14.802168 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-b54v5" podStartSLOduration=2.423905021 podStartE2EDuration="10.802150469s" totalImagesPullingTime="8.378245448s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:04:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:14.801959134 +0000 UTC m=+11.744651978" watchObservedRunningTime="2026-10-30 00:04:14.802150469 +0000 UTC m=+11.744843307" Oct 30 00:04:14.802901 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:14.802839 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-56lxt" podStartSLOduration=2.619994413 podStartE2EDuration="10.802827867s" totalImagesPullingTime="8.182833454s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:04:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:14.764850701 +0000 UTC m=+11.707543543" watchObservedRunningTime="2026-10-30 00:04:14.802827867 +0000 UTC m=+11.745520710" Oct 30 00:04:15.594020 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:15.593986 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxvqx" Oct 30 00:04:15.594553 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:15.594090 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxvqx" podUID="a20efc24-e4cb-4075-911e-bfc43dcfc20b" Oct 30 00:04:15.594553 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:15.594438 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:15.594553 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:15.594527 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zm28x" podUID="fb8917db-c1ba-43af-995e-1948de69aa29" Oct 30 00:04:15.594686 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:15.594576 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:15.594686 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:15.594642 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-52qpr" podUID="30673bf9-d332-4592-afce-b16db49876cb" Oct 30 00:04:15.737091 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:15.737006 2580 generic.go:356] "Generic (PLEG): container finished" podID="0440635e-2cf5-4f40-a4f5-e716376044ba" containerID="1e56847d6524bcf16661d1b2c977c86d67a72f938ed7947dca9258561e5db08a" exitCode=0 Oct 30 00:04:15.737222 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:15.737120 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l54xc" event={"ID":"0440635e-2cf5-4f40-a4f5-e716376044ba","Type":"ContainerDied","Data":"1e56847d6524bcf16661d1b2c977c86d67a72f938ed7947dca9258561e5db08a"} Oct 30 00:04:15.739722 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:15.739699 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j6chh" event={"ID":"10514d09-1dbf-47c2-bc1b-54a15f384cb3","Type":"ContainerStarted","Data":"5c48d3660a01de076c77919d4145ae85e0aad4ef26c635142ab1a75c5bcfb36c"} Oct 30 00:04:15.742751 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:15.742726 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" event={"ID":"31c2770e-628c-4bcd-a088-73f190c952cd","Type":"ContainerStarted","Data":"a696513d61a7f7ccb323cd676ae5ae4430ea01269d684b70152c1f146e50d856"} Oct 30 00:04:15.814877 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:15.814819 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-j6chh" podStartSLOduration=3.640948226 podStartE2EDuration="11.814805883s" totalImagesPullingTime="8.173857657s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:04:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:15.814697048 +0000 UTC m=+12.757389893" watchObservedRunningTime="2026-10-30 00:04:15.814805883 +0000 UTC m=+12.757498725" Oct 30 00:04:16.747813 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:16.747737 2580 generic.go:356] "Generic (PLEG): container finished" podID="0440635e-2cf5-4f40-a4f5-e716376044ba" containerID="97b4a76bd4f36ce5db8e919c83c8f46d42a090ea4dec95ad8df3158128f36946" exitCode=0 Oct 30 00:04:16.747813 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:16.747779 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l54xc" event={"ID":"0440635e-2cf5-4f40-a4f5-e716376044ba","Type":"ContainerDied","Data":"97b4a76bd4f36ce5db8e919c83c8f46d42a090ea4dec95ad8df3158128f36946"} Oct 30 00:04:16.749629 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:16.749601 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-575b9" event={"ID":"0fed8e88-bc49-4564-abcc-7d0fff068aa4","Type":"ContainerStarted","Data":"aefa516750804f0f34c8fa6b371329d64849a96554c58c0f78d1933b31b0a665"} Oct 30 00:04:16.790360 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:16.790304 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" podStartSLOduration=4.009023936 podStartE2EDuration="12.790286228s" totalImagesPullingTime="8.781262292s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:04:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:15.853339605 +0000 UTC m=+12.796032448" watchObservedRunningTime="2026-10-30 00:04:16.790286228 +0000 UTC m=+13.732979074" Oct 30 00:04:16.831294 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:16.831245 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-575b9" podStartSLOduration=2.247484293 podStartE2EDuration="12.831232848s" totalImagesPullingTime="10.583748555s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:04:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:16.831116739 +0000 UTC m=+13.773809581" watchObservedRunningTime="2026-10-30 00:04:16.831232848 +0000 UTC m=+13.773925681" Oct 30 00:04:17.591354 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:17.591315 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:17.591514 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:17.591445 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zm28x" podUID="fb8917db-c1ba-43af-995e-1948de69aa29" Oct 30 00:04:17.591514 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:17.591485 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:17.591654 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:17.591529 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxvqx" Oct 30 00:04:17.591654 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:17.591606 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxvqx" podUID="a20efc24-e4cb-4075-911e-bfc43dcfc20b" Oct 30 00:04:17.591757 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:17.591693 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-52qpr" podUID="30673bf9-d332-4592-afce-b16db49876cb" Oct 30 00:04:18.757562 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:18.757360 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l54xc" event={"ID":"0440635e-2cf5-4f40-a4f5-e716376044ba","Type":"ContainerStarted","Data":"f85a053acae262852b776bbc965ee2e7bf3055bc372018f74a4f39475502e1dd"} Oct 30 00:04:19.591998 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:19.591969 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:19.592235 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:19.591969 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:19.592235 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:19.592100 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zm28x" podUID="fb8917db-c1ba-43af-995e-1948de69aa29" Oct 30 00:04:19.592235 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:19.592199 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-52qpr" podUID="30673bf9-d332-4592-afce-b16db49876cb" Oct 30 00:04:19.592235 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:19.591969 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxvqx" Oct 30 00:04:19.592406 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:19.592325 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxvqx" podUID="a20efc24-e4cb-4075-911e-bfc43dcfc20b" Oct 30 00:04:19.762685 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:19.762654 2580 generic.go:356] "Generic (PLEG): container finished" podID="0440635e-2cf5-4f40-a4f5-e716376044ba" containerID="f85a053acae262852b776bbc965ee2e7bf3055bc372018f74a4f39475502e1dd" exitCode=0 Oct 30 00:04:19.763096 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:19.762699 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l54xc" event={"ID":"0440635e-2cf5-4f40-a4f5-e716376044ba","Type":"ContainerDied","Data":"f85a053acae262852b776bbc965ee2e7bf3055bc372018f74a4f39475502e1dd"} Oct 30 00:04:20.767269 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:20.767233 2580 generic.go:356] "Generic (PLEG): container finished" podID="0440635e-2cf5-4f40-a4f5-e716376044ba" containerID="ef109cd5a18003bdc13d98e238f9a49667fee337c40c353ac5dd8ffd7b7937b0" exitCode=0 Oct 30 00:04:20.767640 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:20.767284 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l54xc" event={"ID":"0440635e-2cf5-4f40-a4f5-e716376044ba","Type":"ContainerDied","Data":"ef109cd5a18003bdc13d98e238f9a49667fee337c40c353ac5dd8ffd7b7937b0"} Oct 30 00:04:21.318341 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:21.318122 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs\") pod \"network-metrics-daemon-52qpr\" (UID: \"30673bf9-d332-4592-afce-b16db49876cb\") " pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:21.318465 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:21.318261 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:04:21.318465 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:21.318400 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs podName:30673bf9-d332-4592-afce-b16db49876cb nodeName:}" failed. No retries permitted until 2026-10-30 00:04:37.318382619 +0000 UTC m=+34.261075453 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs") pod "network-metrics-daemon-52qpr" (UID: "30673bf9-d332-4592-afce-b16db49876cb") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:04:21.418578 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:21.418544 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpg2j\" (UniqueName: \"kubernetes.io/projected/a20efc24-e4cb-4075-911e-bfc43dcfc20b-kube-api-access-dpg2j\") pod \"network-check-target-sxvqx\" (UID: \"a20efc24-e4cb-4075-911e-bfc43dcfc20b\") " pod="openshift-network-diagnostics/network-check-target-sxvqx" Oct 30 00:04:21.418707 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:21.418606 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb8917db-c1ba-43af-995e-1948de69aa29-original-pull-secret\") pod \"global-pull-secret-syncer-zm28x\" (UID: \"fb8917db-c1ba-43af-995e-1948de69aa29\") " pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:21.418707 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:21.418695 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:04:21.418774 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:21.418744 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb8917db-c1ba-43af-995e-1948de69aa29-original-pull-secret podName:fb8917db-c1ba-43af-995e-1948de69aa29 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:37.418730545 +0000 UTC m=+34.361423365 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb8917db-c1ba-43af-995e-1948de69aa29-original-pull-secret") pod "global-pull-secret-syncer-zm28x" (UID: "fb8917db-c1ba-43af-995e-1948de69aa29") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:04:21.418774 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:21.418698 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:04:21.418871 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:21.418787 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:04:21.418871 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:21.418800 2580 projected.go:196] Error preparing data for projected volume kube-api-access-dpg2j for pod openshift-network-diagnostics/network-check-target-sxvqx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:04:21.418871 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:21.418841 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a20efc24-e4cb-4075-911e-bfc43dcfc20b-kube-api-access-dpg2j podName:a20efc24-e4cb-4075-911e-bfc43dcfc20b nodeName:}" failed. No retries permitted until 2026-10-30 00:04:37.418829413 +0000 UTC m=+34.361522251 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dpg2j" (UniqueName: "kubernetes.io/projected/a20efc24-e4cb-4075-911e-bfc43dcfc20b-kube-api-access-dpg2j") pod "network-check-target-sxvqx" (UID: "a20efc24-e4cb-4075-911e-bfc43dcfc20b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:04:21.593863 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:21.593794 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxvqx" Oct 30 00:04:21.594001 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:21.593894 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:21.594001 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:21.593930 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:21.594001 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:21.593923 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxvqx" podUID="a20efc24-e4cb-4075-911e-bfc43dcfc20b" Oct 30 00:04:21.594094 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:21.593998 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zm28x" podUID="fb8917db-c1ba-43af-995e-1948de69aa29" Oct 30 00:04:21.594094 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:21.594057 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-52qpr" podUID="30673bf9-d332-4592-afce-b16db49876cb" Oct 30 00:04:21.772486 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:21.772456 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" event={"ID":"2277c280-7906-4c42-87f1-7941012a2d31","Type":"ContainerStarted","Data":"dcfb07075ccde5e63fb6c1b2bc47d0ce19b68c74c9e14b93668ed75f1596bd23"} Oct 30 00:04:21.773009 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:21.772530 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" event={"ID":"2277c280-7906-4c42-87f1-7941012a2d31","Type":"ContainerStarted","Data":"253d69e6d8209346f59cf9a43b0a81f0c76b3d9536c57288a19b657003340594"} Oct 30 00:04:21.773009 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:21.772566 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" event={"ID":"2277c280-7906-4c42-87f1-7941012a2d31","Type":"ContainerStarted","Data":"331663e0236aaf74ef0aba8502f85c9e39da169aa547566d7cb3a85544e5ae0b"} Oct 30 00:04:21.773009 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:21.772606 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" event={"ID":"2277c280-7906-4c42-87f1-7941012a2d31","Type":"ContainerStarted","Data":"46a63c81019894db1fc50a23ad5cca5538375afb551b03f8c2c0cfef34b2c5af"} Oct 30 00:04:21.773009 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:21.772640 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" event={"ID":"2277c280-7906-4c42-87f1-7941012a2d31","Type":"ContainerStarted","Data":"cd8c03133fbf5ca6195bb476309cc6ec643300a30f5d79694b77ce425bee0dec"} Oct 30 00:04:21.773009 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:21.772678 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" event={"ID":"2277c280-7906-4c42-87f1-7941012a2d31","Type":"ContainerStarted","Data":"5af5063ed18e7f7f771ec64935954276ec5a9efeeaf36ef2c82290ed5212485e"} Oct 30 00:04:21.775630 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:21.775604 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l54xc" event={"ID":"0440635e-2cf5-4f40-a4f5-e716376044ba","Type":"ContainerStarted","Data":"d42c326f40781945cadb2f3666b2a44e20328e2bc92117dffabf191b2a142beb"} Oct 30 00:04:21.818537 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:21.818485 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-l54xc" podStartSLOduration=7.201975177 podStartE2EDuration="17.818474327s" totalImagesPullingTime="10.61649915s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:04:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:21.817653 +0000 UTC m=+18.760345841" watchObservedRunningTime="2026-10-30 00:04:21.818474327 +0000 UTC m=+18.761167168" Oct 30 00:04:22.014382 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:22.014317 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-q8m2x" Oct 30 00:04:22.014502 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:22.014472 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 00:04:22.014983 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:22.014964 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-q8m2x" Oct 30 00:04:23.593659 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:23.593590 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:23.593659 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:23.593627 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxvqx" Oct 30 00:04:23.594134 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:23.593706 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zm28x" podUID="fb8917db-c1ba-43af-995e-1948de69aa29" Oct 30 00:04:23.594134 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:23.593736 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxvqx" podUID="a20efc24-e4cb-4075-911e-bfc43dcfc20b" Oct 30 00:04:23.594134 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:23.593793 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:23.594134 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:23.593868 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-52qpr" podUID="30673bf9-d332-4592-afce-b16db49876cb" Oct 30 00:04:23.782554 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:23.782520 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" event={"ID":"2277c280-7906-4c42-87f1-7941012a2d31","Type":"ContainerStarted","Data":"4266770737faa13be1b73a3944172ca98cc8b82f4e2eb515b87a70a98b60b162"} Oct 30 00:04:25.592093 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:25.592023 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:25.592594 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:25.592027 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:25.592594 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:25.592146 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-52qpr" podUID="30673bf9-d332-4592-afce-b16db49876cb" Oct 30 00:04:25.592594 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:25.592023 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxvqx" Oct 30 00:04:25.592594 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:25.592241 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zm28x" podUID="fb8917db-c1ba-43af-995e-1948de69aa29" Oct 30 00:04:25.592594 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:25.592279 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxvqx" podUID="a20efc24-e4cb-4075-911e-bfc43dcfc20b" Oct 30 00:04:25.790626 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:25.790592 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" event={"ID":"2277c280-7906-4c42-87f1-7941012a2d31","Type":"ContainerStarted","Data":"cbd13382182bcfbffdd78575f05b6b8b0c6a0150217d0612ce14dd21f86ed675"} Oct 30 00:04:25.790846 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:25.790830 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:25.806048 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:25.805877 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:25.874143 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:25.874049 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" podStartSLOduration=6.815724158 podStartE2EDuration="21.874037217s" totalImagesPullingTime="15.058313059s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:04:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:25.87380271 +0000 UTC m=+22.816495551" watchObservedRunningTime="2026-10-30 00:04:25.874037217 +0000 UTC m=+22.816730059" Oct 30 00:04:26.793066 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:26.793041 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 00:04:26.793471 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:26.793454 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:26.807953 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:26.807929 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:27.452861 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:27.452832 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zm28x"] Oct 30 00:04:27.453431 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:27.453387 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:27.453523 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:27.453495 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zm28x" podUID="fb8917db-c1ba-43af-995e-1948de69aa29" Oct 30 00:04:27.455761 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:27.455741 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-52qpr"] Oct 30 00:04:27.455854 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:27.455843 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:27.455956 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:27.455941 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-52qpr" podUID="30673bf9-d332-4592-afce-b16db49876cb" Oct 30 00:04:27.456239 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:27.456224 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sxvqx"] Oct 30 00:04:27.456311 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:27.456301 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxvqx" Oct 30 00:04:27.456384 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:27.456368 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxvqx" podUID="a20efc24-e4cb-4075-911e-bfc43dcfc20b" Oct 30 00:04:27.794894 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:27.794871 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 00:04:28.591661 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:28.591634 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxvqx" Oct 30 00:04:28.591820 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:28.591723 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxvqx" podUID="a20efc24-e4cb-4075-911e-bfc43dcfc20b" Oct 30 00:04:28.797056 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:28.797025 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 00:04:29.011383 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:29.011359 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:29.591595 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:29.591564 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:29.591740 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:29.591582 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:29.591740 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:29.591704 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zm28x" podUID="fb8917db-c1ba-43af-995e-1948de69aa29" Oct 30 00:04:29.591813 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:29.591738 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-52qpr" podUID="30673bf9-d332-4592-afce-b16db49876cb" Oct 30 00:04:30.591528 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:30.591487 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxvqx" Oct 30 00:04:30.592078 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:30.591596 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxvqx" podUID="a20efc24-e4cb-4075-911e-bfc43dcfc20b" Oct 30 00:04:31.591843 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:31.591807 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:31.592196 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:31.591846 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:31.592196 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:31.591968 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zm28x" podUID="fb8917db-c1ba-43af-995e-1948de69aa29" Oct 30 00:04:31.592196 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:31.592081 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-52qpr" podUID="30673bf9-d332-4592-afce-b16db49876cb" Oct 30 00:04:32.591168 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:32.591130 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxvqx" Oct 30 00:04:32.591332 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:32.591234 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxvqx" podUID="a20efc24-e4cb-4075-911e-bfc43dcfc20b" Oct 30 00:04:33.592530 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.592500 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:33.592889 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:33.592579 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zm28x" podUID="fb8917db-c1ba-43af-995e-1948de69aa29" Oct 30 00:04:33.592889 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.592683 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:33.592889 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:33.592811 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-52qpr" podUID="30673bf9-d332-4592-afce-b16db49876cb" Oct 30 00:04:33.862752 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.862722 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeReady" Oct 30 00:04:33.862931 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.862858 2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 00:04:33.912879 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.912852 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-m98x7"] Oct 30 00:04:33.925971 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.925946 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-b9fcp"] Oct 30 00:04:33.926083 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.926057 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-m98x7" Oct 30 00:04:33.931889 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.931860 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4fpvt"] Oct 30 00:04:33.932056 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.932017 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-b9fcp" Oct 30 00:04:33.935827 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.935805 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c8c8757f-tspqd"] Oct 30 00:04:33.935969 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.935943 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-4fpvt" Oct 30 00:04:33.938851 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.938833 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-z8gb5"] Oct 30 00:04:33.939100 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.938973 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:33.941783 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.941735 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l25ts"] Oct 30 00:04:33.941903 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.941888 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" Oct 30 00:04:33.944835 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.944816 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-nzxzl"] Oct 30 00:04:33.945004 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.944979 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l25ts" Oct 30 00:04:33.947801 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.947781 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 00:04:33.947801 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.947797 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 00:04:33.948333 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.948315 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-7m2x2"] Oct 30 00:04:33.948441 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.948426 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-nzxzl" Oct 30 00:04:33.950702 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.950685 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-nqw98"] Oct 30 00:04:33.950796 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.950782 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-7m2x2" Oct 30 00:04:33.954364 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.953759 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-n78wz"] Oct 30 00:04:33.956414 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.954510 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nqw98" Oct 30 00:04:33.959600 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.959055 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 00:04:33.960686 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.960617 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 00:04:33.962233 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.962076 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 00:04:33.962233 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.962191 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-smkfn\"" Oct 30 00:04:33.968968 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.968948 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-glf29"] Oct 30 00:04:33.969198 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.969180 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-n78wz" Oct 30 00:04:33.973875 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.973859 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7dcddbb874-9zx5v"] Oct 30 00:04:33.974072 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.974056 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-glf29" Oct 30 00:04:33.979688 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.979428 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-645c8fd54c-hp29d"] Oct 30 00:04:33.979946 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.979926 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:33.984323 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.983589 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-6d24p"] Oct 30 00:04:33.986713 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:33.986626 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:34.000888 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.000866 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l"] Oct 30 00:04:34.001117 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.001100 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6d24p" Oct 30 00:04:34.004305 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.004287 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-52kw4"] Oct 30 00:04:34.004420 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.004398 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l" Oct 30 00:04:34.007478 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.007462 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-m98x7"] Oct 30 00:04:34.007550 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.007488 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mpg4t"] Oct 30 00:04:34.007586 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.007553 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-52kw4" Oct 30 00:04:34.009996 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.009978 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddgh4\" (UniqueName: \"kubernetes.io/projected/6f2b3396-a95d-4bc7-902f-1daf3dd689fd-kube-api-access-ddgh4\") pod \"kas-connection-checker-685895c7c9-b9fcp\" (UID: \"6f2b3396-a95d-4bc7-902f-1daf3dd689fd\") " pod="kube-system/kas-connection-checker-685895c7c9-b9fcp" Oct 30 00:04:34.010075 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.010027 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4snt\" (UniqueName: \"kubernetes.io/projected/c3ce5e93-7d27-4fa6-b360-e571d8f18481-kube-api-access-v4snt\") pod \"kas-connection-checker-685895c7c9-4fpvt\" (UID: \"c3ce5e93-7d27-4fa6-b360-e571d8f18481\") " pod="kube-system/kas-connection-checker-685895c7c9-4fpvt" Oct 30 00:04:34.010117 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.010093 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spcsf\" (UniqueName: \"kubernetes.io/projected/d2413398-197b-4cad-8dc4-9bba9effe3b7-kube-api-access-spcsf\") pod \"kas-connection-checker-685895c7c9-m98x7\" (UID: \"d2413398-197b-4cad-8dc4-9bba9effe3b7\") " pod="kube-system/kas-connection-checker-685895c7c9-m98x7" Oct 30 00:04:34.010876 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.010857 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-b9fcp"] Oct 30 00:04:34.010972 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.010897 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4fpvt"] Oct 30 00:04:34.010972 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.010938 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-7m2x2"] Oct 30 00:04:34.010972 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.010962 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l25ts"] Oct 30 00:04:34.011386 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.010991 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-nzxzl"] Oct 30 00:04:34.011386 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.011012 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c8c8757f-tspqd"] Oct 30 00:04:34.011386 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.011033 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-glf29"] Oct 30 00:04:34.011386 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.011055 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mpg4t"] Oct 30 00:04:34.011386 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.011080 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l"] Oct 30 00:04:34.011386 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.011101 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-n78wz"] Oct 30 00:04:34.011386 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.011121 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-6d24p"] Oct 30 00:04:34.011386 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.011147 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-z8gb5"] Oct 30 00:04:34.011386 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.011153 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mpg4t" Oct 30 00:04:34.011386 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.011166 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-nqw98"] Oct 30 00:04:34.011386 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.011188 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-645c8fd54c-hp29d"] Oct 30 00:04:34.011386 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.011209 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7dcddbb874-9zx5v"] Oct 30 00:04:34.011386 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.011225 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-52kw4"] Oct 30 00:04:34.048036 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.048009 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 00:04:34.048140 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.048095 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 00:04:34.048140 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.048113 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:04:34.048140 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.048125 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 00:04:34.048267 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.048147 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 00:04:34.048267 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.048190 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:04:34.048267 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.048209 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:04:34.048267 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.048244 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 00:04:34.048267 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.048247 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 00:04:34.048455 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.048327 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 00:04:34.048455 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.048395 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 00:04:34.048455 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.048418 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 00:04:34.048577 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.048498 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 00:04:34.048577 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.048525 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 00:04:34.048660 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.048584 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 00:04:34.048660 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.048628 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 00:04:34.048742 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.048699 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 00:04:34.048775 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.048769 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 00:04:34.048826 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.048778 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 00:04:34.048826 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.048792 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 00:04:34.048826 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.048818 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:04:34.048826 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.048826 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 00:04:34.048979 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.048828 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 00:04:34.048979 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.048824 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 00:04:34.048979 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.048783 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 00:04:34.048979 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.048859 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 00:04:34.048979 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.048876 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:04:34.061290 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.061276 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 00:04:34.061686 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.061674 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 00:04:34.074946 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.074926 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 00:04:34.075270 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.075047 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 00:04:34.075270 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.075059 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 00:04:34.075270 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.075068 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 00:04:34.075270 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.075078 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 00:04:34.075270 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.075148 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 00:04:34.075270 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.075167 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 00:04:34.075270 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.075187 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 00:04:34.075270 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.075203 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 00:04:34.075270 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.075233 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 00:04:34.075270 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.075240 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 00:04:34.075270 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.075240 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 00:04:34.075270 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.075234 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 00:04:34.075835 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.075434 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9vr82\"" Oct 30 00:04:34.075835 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.075435 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-cmxwh\"" Oct 30 00:04:34.075835 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.075473 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-lqg85\"" Oct 30 00:04:34.075835 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.075483 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ntcgl\"" Oct 30 00:04:34.075835 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.075521 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-mjr9m\"" Oct 30 00:04:34.075835 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.075582 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 00:04:34.076174 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.076155 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-j84nm\"" Oct 30 00:04:34.077013 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.076993 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 00:04:34.110852 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.110824 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9fa688d8-29ca-4c1f-926a-f1a4099e6b44-serving-cert\") pod \"console-operator-7cd56b8dbc-z8gb5\" (UID: \"9fa688d8-29ca-4c1f-926a-f1a4099e6b44\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" Oct 30 00:04:34.110852 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.110855 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-metrics-certs\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:34.111071 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.110873 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-installation-pull-secrets\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:34.111071 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.110891 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-default-certificate\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:34.111071 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.110907 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-serving-cert\") pod \"insights-operator-7c4986d449-nzxzl\" (UID: \"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8\") " pod="openshift-insights/insights-operator-7c4986d449-nzxzl" Oct 30 00:04:34.111071 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.111010 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jzxf\" (UniqueName: \"kubernetes.io/projected/4350294b-64f3-4efd-9619-90c013d82432-kube-api-access-7jzxf\") pod \"volume-data-source-validator-58c9f5f46d-n78wz\" (UID: \"4350294b-64f3-4efd-9619-90c013d82432\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-n78wz" Oct 30 00:04:34.111211 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.111069 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-registry-tls\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:34.111211 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.111095 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xrgq\" (UniqueName: \"kubernetes.io/projected/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-kube-api-access-6xrgq\") pod \"insights-operator-7c4986d449-nzxzl\" (UID: \"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8\") " pod="openshift-insights/insights-operator-7c4986d449-nzxzl" Oct 30 00:04:34.111211 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.111114 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-bound-sa-token\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:34.111211 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.111133 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd5wb\" (UniqueName: \"kubernetes.io/projected/8a82abd0-5057-4361-9b0d-7a4f23ae97d1-kube-api-access-xd5wb\") pod \"service-ca-operator-544d749c8c-6d24p\" (UID: \"8a82abd0-5057-4361-9b0d-7a4f23ae97d1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6d24p" Oct 30 00:04:34.111211 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.111191 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/526cc561-1bbd-4079-8a2f-326f939753e0-installation-pull-secrets\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:34.111211 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.111210 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/aff3a44b-6cad-449e-8733-f85e9899ecc3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-nqw98\" (UID: \"aff3a44b-6cad-449e-8733-f85e9899ecc3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nqw98" Oct 30 00:04:34.111424 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.111228 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddgh4\" (UniqueName: \"kubernetes.io/projected/6f2b3396-a95d-4bc7-902f-1daf3dd689fd-kube-api-access-ddgh4\") pod \"kas-connection-checker-685895c7c9-b9fcp\" (UID: \"6f2b3396-a95d-4bc7-902f-1daf3dd689fd\") " pod="kube-system/kas-connection-checker-685895c7c9-b9fcp" Oct 30 00:04:34.111424 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.111308 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9fa688d8-29ca-4c1f-926a-f1a4099e6b44-trusted-ca\") pod \"console-operator-7cd56b8dbc-z8gb5\" (UID: \"9fa688d8-29ca-4c1f-926a-f1a4099e6b44\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" Oct 30 00:04:34.111424 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.111366 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8xpk\" (UniqueName: \"kubernetes.io/projected/18a4ebbe-c390-49d8-99e7-858bfaa3b834-kube-api-access-s8xpk\") pod \"cluster-samples-operator-6ddb6c8b94-l25ts\" (UID: \"18a4ebbe-c390-49d8-99e7-858bfaa3b834\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l25ts" Oct 30 00:04:34.111424 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.111411 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/acc82f37-fad2-424d-89d0-5e5876cb75de-config-volume\") pod \"dns-default-52kw4\" (UID: \"acc82f37-fad2-424d-89d0-5e5876cb75de\") " pod="openshift-dns/dns-default-52kw4" Oct 30 00:04:34.111550 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.111448 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dpbj\" (UniqueName: \"kubernetes.io/projected/32309642-7cba-4361-83b3-4680659c0544-kube-api-access-5dpbj\") pod \"network-check-source-6f78d59f99-glf29\" (UID: \"32309642-7cba-4361-83b3-4680659c0544\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-glf29" Oct 30 00:04:34.111550 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.111508 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acc82f37-fad2-424d-89d0-5e5876cb75de-metrics-tls\") pod \"dns-default-52kw4\" (UID: \"acc82f37-fad2-424d-89d0-5e5876cb75de\") " pod="openshift-dns/dns-default-52kw4" Oct 30 00:04:34.111550 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.111531 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cngbs\" (UniqueName: \"kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-kube-api-access-cngbs\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:34.111687 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.111554 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-l25ts\" (UID: \"18a4ebbe-c390-49d8-99e7-858bfaa3b834\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l25ts" Oct 30 00:04:34.111687 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.111589 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-image-registry-private-configuration\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:34.111687 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.111617 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f87wl\" (UniqueName: \"kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-kube-api-access-f87wl\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:34.111687 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.111635 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1a647f1-e715-4a8a-8b44-0196b3b1af27-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l\" (UID: \"c1a647f1-e715-4a8a-8b44-0196b3b1af27\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l" Oct 30 00:04:34.111687 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.111654 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-bound-sa-token\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:34.111884 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.111703 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-trusted-ca\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:34.111884 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.111722 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spcsf\" (UniqueName: \"kubernetes.io/projected/d2413398-197b-4cad-8dc4-9bba9effe3b7-kube-api-access-spcsf\") pod \"kas-connection-checker-685895c7c9-m98x7\" (UID: \"d2413398-197b-4cad-8dc4-9bba9effe3b7\") " pod="kube-system/kas-connection-checker-685895c7c9-m98x7" Oct 30 00:04:34.111884 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.111746 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mglpv\" (UniqueName: \"kubernetes.io/projected/de19ca9c-b22d-49a4-a5a6-a5f3a61bf892-kube-api-access-mglpv\") pod \"ingress-canary-mpg4t\" (UID: \"de19ca9c-b22d-49a4-a5a6-a5f3a61bf892\") " pod="openshift-ingress-canary/ingress-canary-mpg4t" Oct 30 00:04:34.111884 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.111770 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-tmp\") pod \"insights-operator-7c4986d449-nzxzl\" (UID: \"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8\") " pod="openshift-insights/insights-operator-7c4986d449-nzxzl" Oct 30 00:04:34.111884 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.111786 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-stats-auth\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:34.111884 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.111801 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/526cc561-1bbd-4079-8a2f-326f939753e0-trusted-ca\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:34.111884 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.111837 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0d3be8a7-773b-42e1-a9a1-60d7df717301-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7m2x2\" (UID: \"0d3be8a7-773b-42e1-a9a1-60d7df717301\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7m2x2" Oct 30 00:04:34.112165 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.111893 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7m2x2\" (UID: \"0d3be8a7-773b-42e1-a9a1-60d7df717301\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7m2x2" Oct 30 00:04:34.112165 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.111936 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/526cc561-1bbd-4079-8a2f-326f939753e0-registry-certificates\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:34.112165 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.111962 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-ca-trust-extracted\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:34.112165 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.111995 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56j8l\" (UniqueName: \"kubernetes.io/projected/aff3a44b-6cad-449e-8733-f85e9899ecc3-kube-api-access-56j8l\") pod \"cluster-monitoring-operator-884576c57-nqw98\" (UID: \"aff3a44b-6cad-449e-8733-f85e9899ecc3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nqw98" Oct 30 00:04:34.112165 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.112042 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a82abd0-5057-4361-9b0d-7a4f23ae97d1-config\") pod \"service-ca-operator-544d749c8c-6d24p\" (UID: \"8a82abd0-5057-4361-9b0d-7a4f23ae97d1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6d24p" Oct 30 00:04:34.112165 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.112106 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l25ts\" (UID: \"18a4ebbe-c390-49d8-99e7-858bfaa3b834\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l25ts" Oct 30 00:04:34.112165 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.112131 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-tls\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:34.112393 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.112168 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acc82f37-fad2-424d-89d0-5e5876cb75de-tmp-dir\") pod \"dns-default-52kw4\" (UID: \"acc82f37-fad2-424d-89d0-5e5876cb75de\") " pod="openshift-dns/dns-default-52kw4" Oct 30 00:04:34.112393 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.112184 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/526cc561-1bbd-4079-8a2f-326f939753e0-image-registry-private-configuration\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:34.112393 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.112215 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ht68\" (UniqueName: \"kubernetes.io/projected/9d080b8e-cca6-4040-830e-cc9e9e25c50a-kube-api-access-6ht68\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:34.112393 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.112239 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aff3a44b-6cad-449e-8733-f85e9899ecc3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nqw98\" (UID: \"aff3a44b-6cad-449e-8733-f85e9899ecc3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nqw98" Oct 30 00:04:34.112393 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.112289 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1a647f1-e715-4a8a-8b44-0196b3b1af27-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l\" (UID: \"c1a647f1-e715-4a8a-8b44-0196b3b1af27\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l" Oct 30 00:04:34.112393 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.112319 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfc4x\" (UniqueName: \"kubernetes.io/projected/acc82f37-fad2-424d-89d0-5e5876cb75de-kube-api-access-kfc4x\") pod \"dns-default-52kw4\" (UID: \"acc82f37-fad2-424d-89d0-5e5876cb75de\") " pod="openshift-dns/dns-default-52kw4" Oct 30 00:04:34.112393 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.112334 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-certificates\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:34.112393 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.112349 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-snapshots\") pod \"insights-operator-7c4986d449-nzxzl\" (UID: \"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8\") " pod="openshift-insights/insights-operator-7c4986d449-nzxzl" Oct 30 00:04:34.112643 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.112401 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cv5p\" (UniqueName: \"kubernetes.io/projected/9fa688d8-29ca-4c1f-926a-f1a4099e6b44-kube-api-access-6cv5p\") pod \"console-operator-7cd56b8dbc-z8gb5\" (UID: \"9fa688d8-29ca-4c1f-926a-f1a4099e6b44\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" Oct 30 00:04:34.112643 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.112435 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4snt\" (UniqueName: \"kubernetes.io/projected/c3ce5e93-7d27-4fa6-b360-e571d8f18481-kube-api-access-v4snt\") pod \"kas-connection-checker-685895c7c9-4fpvt\" (UID: \"c3ce5e93-7d27-4fa6-b360-e571d8f18481\") " pod="kube-system/kas-connection-checker-685895c7c9-4fpvt" Oct 30 00:04:34.112643 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.112451 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-service-ca-bundle\") pod \"insights-operator-7c4986d449-nzxzl\" (UID: \"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8\") " pod="openshift-insights/insights-operator-7c4986d449-nzxzl" Oct 30 00:04:34.112643 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.112505 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d080b8e-cca6-4040-830e-cc9e9e25c50a-service-ca-bundle\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:34.112643 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.112536 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v899\" (UniqueName: \"kubernetes.io/projected/c1a647f1-e715-4a8a-8b44-0196b3b1af27-kube-api-access-2v899\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l\" (UID: \"c1a647f1-e715-4a8a-8b44-0196b3b1af27\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l" Oct 30 00:04:34.112643 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.112578 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a82abd0-5057-4361-9b0d-7a4f23ae97d1-serving-cert\") pod \"service-ca-operator-544d749c8c-6d24p\" (UID: \"8a82abd0-5057-4361-9b0d-7a4f23ae97d1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6d24p" Oct 30 00:04:34.112643 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.112629 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de19ca9c-b22d-49a4-a5a6-a5f3a61bf892-cert\") pod \"ingress-canary-mpg4t\" (UID: \"de19ca9c-b22d-49a4-a5a6-a5f3a61bf892\") " pod="openshift-ingress-canary/ingress-canary-mpg4t" Oct 30 00:04:34.112845 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.112655 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-nzxzl\" (UID: \"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8\") " pod="openshift-insights/insights-operator-7c4986d449-nzxzl" Oct 30 00:04:34.112845 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.112673 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/526cc561-1bbd-4079-8a2f-326f939753e0-ca-trust-extracted\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:34.112845 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.112688 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9fa688d8-29ca-4c1f-926a-f1a4099e6b44-config\") pod \"console-operator-7cd56b8dbc-z8gb5\" (UID: \"9fa688d8-29ca-4c1f-926a-f1a4099e6b44\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" Oct 30 00:04:34.135832 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.135804 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddgh4\" (UniqueName: \"kubernetes.io/projected/6f2b3396-a95d-4bc7-902f-1daf3dd689fd-kube-api-access-ddgh4\") pod \"kas-connection-checker-685895c7c9-b9fcp\" (UID: \"6f2b3396-a95d-4bc7-902f-1daf3dd689fd\") " pod="kube-system/kas-connection-checker-685895c7c9-b9fcp" Oct 30 00:04:34.135832 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.135812 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spcsf\" (UniqueName: \"kubernetes.io/projected/d2413398-197b-4cad-8dc4-9bba9effe3b7-kube-api-access-spcsf\") pod \"kas-connection-checker-685895c7c9-m98x7\" (UID: \"d2413398-197b-4cad-8dc4-9bba9effe3b7\") " pod="kube-system/kas-connection-checker-685895c7c9-m98x7" Oct 30 00:04:34.135832 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.135805 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4snt\" (UniqueName: \"kubernetes.io/projected/c3ce5e93-7d27-4fa6-b360-e571d8f18481-kube-api-access-v4snt\") pod \"kas-connection-checker-685895c7c9-4fpvt\" (UID: \"c3ce5e93-7d27-4fa6-b360-e571d8f18481\") " pod="kube-system/kas-connection-checker-685895c7c9-4fpvt" Oct 30 00:04:34.213700 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.213669 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l25ts\" (UID: \"18a4ebbe-c390-49d8-99e7-858bfaa3b834\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l25ts" Oct 30 00:04:34.213700 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.213701 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-tls\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:34.213889 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.213723 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acc82f37-fad2-424d-89d0-5e5876cb75de-tmp-dir\") pod \"dns-default-52kw4\" (UID: \"acc82f37-fad2-424d-89d0-5e5876cb75de\") " pod="openshift-dns/dns-default-52kw4" Oct 30 00:04:34.213889 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.213751 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/526cc561-1bbd-4079-8a2f-326f939753e0-image-registry-private-configuration\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:34.213889 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.213791 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ht68\" (UniqueName: \"kubernetes.io/projected/9d080b8e-cca6-4040-830e-cc9e9e25c50a-kube-api-access-6ht68\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:34.213889 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.213816 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aff3a44b-6cad-449e-8733-f85e9899ecc3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nqw98\" (UID: \"aff3a44b-6cad-449e-8733-f85e9899ecc3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nqw98" Oct 30 00:04:34.213889 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.213846 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1a647f1-e715-4a8a-8b44-0196b3b1af27-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l\" (UID: \"c1a647f1-e715-4a8a-8b44-0196b3b1af27\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l" Oct 30 00:04:34.213889 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.213818 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:04:34.213889 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.213880 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfc4x\" (UniqueName: \"kubernetes.io/projected/acc82f37-fad2-424d-89d0-5e5876cb75de-kube-api-access-kfc4x\") pod \"dns-default-52kw4\" (UID: \"acc82f37-fad2-424d-89d0-5e5876cb75de\") " pod="openshift-dns/dns-default-52kw4" Oct 30 00:04:34.213889 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.213886 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c8c8757f-tspqd: secret "image-registry-tls" not found Oct 30 00:04:34.214307 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.213942 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 00:04:34.214307 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.213984 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-tls podName:fd8f9a4a-5b77-4a52-9ea9-a12245c00489 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:34.71396261 +0000 UTC m=+31.656655453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-tls") pod "image-registry-7c8c8757f-tspqd" (UID: "fd8f9a4a-5b77-4a52-9ea9-a12245c00489") : secret "image-registry-tls" not found Oct 30 00:04:34.214307 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.213822 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 00:04:34.214307 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.214011 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff3a44b-6cad-449e-8733-f85e9899ecc3-cluster-monitoring-operator-tls podName:aff3a44b-6cad-449e-8733-f85e9899ecc3 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:34.713992273 +0000 UTC m=+31.656685106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/aff3a44b-6cad-449e-8733-f85e9899ecc3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nqw98" (UID: "aff3a44b-6cad-449e-8733-f85e9899ecc3") : secret "cluster-monitoring-operator-tls" not found Oct 30 00:04:34.214307 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214043 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-certificates\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:34.214307 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.214048 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-tls podName:18a4ebbe-c390-49d8-99e7-858bfaa3b834 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:34.714037484 +0000 UTC m=+31.656730324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-l25ts" (UID: "18a4ebbe-c390-49d8-99e7-858bfaa3b834") : secret "samples-operator-tls" not found Oct 30 00:04:34.214307 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214082 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-snapshots\") pod \"insights-operator-7c4986d449-nzxzl\" (UID: \"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8\") " pod="openshift-insights/insights-operator-7c4986d449-nzxzl" Oct 30 00:04:34.214307 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214093 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acc82f37-fad2-424d-89d0-5e5876cb75de-tmp-dir\") pod \"dns-default-52kw4\" (UID: \"acc82f37-fad2-424d-89d0-5e5876cb75de\") " pod="openshift-dns/dns-default-52kw4" Oct 30 00:04:34.214307 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214108 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cv5p\" (UniqueName: \"kubernetes.io/projected/9fa688d8-29ca-4c1f-926a-f1a4099e6b44-kube-api-access-6cv5p\") pod \"console-operator-7cd56b8dbc-z8gb5\" (UID: \"9fa688d8-29ca-4c1f-926a-f1a4099e6b44\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" Oct 30 00:04:34.214307 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214151 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-service-ca-bundle\") pod \"insights-operator-7c4986d449-nzxzl\" (UID: \"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8\") " pod="openshift-insights/insights-operator-7c4986d449-nzxzl" Oct 30 00:04:34.214307 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214181 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d080b8e-cca6-4040-830e-cc9e9e25c50a-service-ca-bundle\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:34.214307 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214207 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v899\" (UniqueName: \"kubernetes.io/projected/c1a647f1-e715-4a8a-8b44-0196b3b1af27-kube-api-access-2v899\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l\" (UID: \"c1a647f1-e715-4a8a-8b44-0196b3b1af27\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l" Oct 30 00:04:34.214307 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214235 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a82abd0-5057-4361-9b0d-7a4f23ae97d1-serving-cert\") pod \"service-ca-operator-544d749c8c-6d24p\" (UID: \"8a82abd0-5057-4361-9b0d-7a4f23ae97d1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6d24p" Oct 30 00:04:34.214307 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214260 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de19ca9c-b22d-49a4-a5a6-a5f3a61bf892-cert\") pod \"ingress-canary-mpg4t\" (UID: \"de19ca9c-b22d-49a4-a5a6-a5f3a61bf892\") " pod="openshift-ingress-canary/ingress-canary-mpg4t" Oct 30 00:04:34.214307 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214285 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-nzxzl\" (UID: \"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8\") " pod="openshift-insights/insights-operator-7c4986d449-nzxzl" Oct 30 00:04:34.215113 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214311 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/526cc561-1bbd-4079-8a2f-326f939753e0-ca-trust-extracted\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:34.215113 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214335 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9fa688d8-29ca-4c1f-926a-f1a4099e6b44-config\") pod \"console-operator-7cd56b8dbc-z8gb5\" (UID: \"9fa688d8-29ca-4c1f-926a-f1a4099e6b44\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" Oct 30 00:04:34.215113 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214364 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9fa688d8-29ca-4c1f-926a-f1a4099e6b44-serving-cert\") pod \"console-operator-7cd56b8dbc-z8gb5\" (UID: \"9fa688d8-29ca-4c1f-926a-f1a4099e6b44\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" Oct 30 00:04:34.215113 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214399 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-metrics-certs\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:34.215113 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214425 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-installation-pull-secrets\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:34.215113 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214450 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-default-certificate\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:34.215113 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214475 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-serving-cert\") pod \"insights-operator-7c4986d449-nzxzl\" (UID: \"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8\") " pod="openshift-insights/insights-operator-7c4986d449-nzxzl" Oct 30 00:04:34.215113 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214510 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jzxf\" (UniqueName: \"kubernetes.io/projected/4350294b-64f3-4efd-9619-90c013d82432-kube-api-access-7jzxf\") pod \"volume-data-source-validator-58c9f5f46d-n78wz\" (UID: \"4350294b-64f3-4efd-9619-90c013d82432\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-n78wz" Oct 30 00:04:34.215113 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214539 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-registry-tls\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:34.215113 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214568 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xrgq\" (UniqueName: \"kubernetes.io/projected/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-kube-api-access-6xrgq\") pod \"insights-operator-7c4986d449-nzxzl\" (UID: \"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8\") " pod="openshift-insights/insights-operator-7c4986d449-nzxzl" Oct 30 00:04:34.215113 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214593 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-bound-sa-token\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:34.215113 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214619 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd5wb\" (UniqueName: \"kubernetes.io/projected/8a82abd0-5057-4361-9b0d-7a4f23ae97d1-kube-api-access-xd5wb\") pod \"service-ca-operator-544d749c8c-6d24p\" (UID: \"8a82abd0-5057-4361-9b0d-7a4f23ae97d1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6d24p" Oct 30 00:04:34.215113 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214651 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/526cc561-1bbd-4079-8a2f-326f939753e0-installation-pull-secrets\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:34.215113 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214676 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/aff3a44b-6cad-449e-8733-f85e9899ecc3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-nqw98\" (UID: \"aff3a44b-6cad-449e-8733-f85e9899ecc3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nqw98" Oct 30 00:04:34.215113 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214701 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9fa688d8-29ca-4c1f-926a-f1a4099e6b44-trusted-ca\") pod \"console-operator-7cd56b8dbc-z8gb5\" (UID: \"9fa688d8-29ca-4c1f-926a-f1a4099e6b44\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" Oct 30 00:04:34.215113 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214732 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8xpk\" (UniqueName: \"kubernetes.io/projected/18a4ebbe-c390-49d8-99e7-858bfaa3b834-kube-api-access-s8xpk\") pod \"cluster-samples-operator-6ddb6c8b94-l25ts\" (UID: \"18a4ebbe-c390-49d8-99e7-858bfaa3b834\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l25ts" Oct 30 00:04:34.215864 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214761 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/acc82f37-fad2-424d-89d0-5e5876cb75de-config-volume\") pod \"dns-default-52kw4\" (UID: \"acc82f37-fad2-424d-89d0-5e5876cb75de\") " pod="openshift-dns/dns-default-52kw4" Oct 30 00:04:34.215864 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214784 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dpbj\" (UniqueName: \"kubernetes.io/projected/32309642-7cba-4361-83b3-4680659c0544-kube-api-access-5dpbj\") pod \"network-check-source-6f78d59f99-glf29\" (UID: \"32309642-7cba-4361-83b3-4680659c0544\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-glf29" Oct 30 00:04:34.215864 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214814 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-certificates\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:34.215864 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214828 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acc82f37-fad2-424d-89d0-5e5876cb75de-metrics-tls\") pod \"dns-default-52kw4\" (UID: \"acc82f37-fad2-424d-89d0-5e5876cb75de\") " pod="openshift-dns/dns-default-52kw4" Oct 30 00:04:34.215864 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214865 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cngbs\" (UniqueName: \"kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-kube-api-access-cngbs\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:34.215864 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214898 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-l25ts\" (UID: \"18a4ebbe-c390-49d8-99e7-858bfaa3b834\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l25ts" Oct 30 00:04:34.215864 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.214907 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:04:34.215864 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214946 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-image-registry-private-configuration\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:34.215864 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214986 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f87wl\" (UniqueName: \"kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-kube-api-access-f87wl\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:34.215864 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.215013 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1a647f1-e715-4a8a-8b44-0196b3b1af27-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l\" (UID: \"c1a647f1-e715-4a8a-8b44-0196b3b1af27\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l" Oct 30 00:04:34.215864 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.215038 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-bound-sa-token\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:34.215864 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.215097 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-trusted-ca\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:34.215864 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.215129 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mglpv\" (UniqueName: \"kubernetes.io/projected/de19ca9c-b22d-49a4-a5a6-a5f3a61bf892-kube-api-access-mglpv\") pod \"ingress-canary-mpg4t\" (UID: \"de19ca9c-b22d-49a4-a5a6-a5f3a61bf892\") " pod="openshift-ingress-canary/ingress-canary-mpg4t" Oct 30 00:04:34.215864 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.215163 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-tmp\") pod \"insights-operator-7c4986d449-nzxzl\" (UID: \"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8\") " pod="openshift-insights/insights-operator-7c4986d449-nzxzl" Oct 30 00:04:34.215864 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.215188 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-stats-auth\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:34.215864 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.215211 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/526cc561-1bbd-4079-8a2f-326f939753e0-trusted-ca\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:34.215864 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.215267 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 00:04:34.216705 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.215318 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-metrics-certs podName:9d080b8e-cca6-4040-830e-cc9e9e25c50a nodeName:}" failed. No retries permitted until 2026-10-30 00:04:34.715302795 +0000 UTC m=+31.657995623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-metrics-certs") pod "router-default-645c8fd54c-hp29d" (UID: "9d080b8e-cca6-4040-830e-cc9e9e25c50a") : secret "router-metrics-certs-default" not found Oct 30 00:04:34.216705 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.215378 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0d3be8a7-773b-42e1-a9a1-60d7df717301-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7m2x2\" (UID: \"0d3be8a7-773b-42e1-a9a1-60d7df717301\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7m2x2" Oct 30 00:04:34.216705 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.215423 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7m2x2\" (UID: \"0d3be8a7-773b-42e1-a9a1-60d7df717301\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7m2x2" Oct 30 00:04:34.216705 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.215454 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/526cc561-1bbd-4079-8a2f-326f939753e0-registry-certificates\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:34.216705 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.215465 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acc82f37-fad2-424d-89d0-5e5876cb75de-metrics-tls podName:acc82f37-fad2-424d-89d0-5e5876cb75de nodeName:}" failed. No retries permitted until 2026-10-30 00:04:34.715448441 +0000 UTC m=+31.658141282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acc82f37-fad2-424d-89d0-5e5876cb75de-metrics-tls") pod "dns-default-52kw4" (UID: "acc82f37-fad2-424d-89d0-5e5876cb75de") : secret "dns-default-metrics-tls" not found Oct 30 00:04:34.216705 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.215507 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-ca-trust-extracted\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:34.216705 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.215540 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56j8l\" (UniqueName: \"kubernetes.io/projected/aff3a44b-6cad-449e-8733-f85e9899ecc3-kube-api-access-56j8l\") pod \"cluster-monitoring-operator-884576c57-nqw98\" (UID: \"aff3a44b-6cad-449e-8733-f85e9899ecc3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nqw98" Oct 30 00:04:34.216705 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.215567 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a82abd0-5057-4361-9b0d-7a4f23ae97d1-config\") pod \"service-ca-operator-544d749c8c-6d24p\" (UID: \"8a82abd0-5057-4361-9b0d-7a4f23ae97d1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6d24p" Oct 30 00:04:34.216705 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.215966 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9fa688d8-29ca-4c1f-926a-f1a4099e6b44-config\") pod \"console-operator-7cd56b8dbc-z8gb5\" (UID: \"9fa688d8-29ca-4c1f-926a-f1a4099e6b44\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" Oct 30 00:04:34.216705 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.216154 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a82abd0-5057-4361-9b0d-7a4f23ae97d1-config\") pod \"service-ca-operator-544d749c8c-6d24p\" (UID: \"8a82abd0-5057-4361-9b0d-7a4f23ae97d1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6d24p" Oct 30 00:04:34.216705 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.216522 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-service-ca-bundle\") pod \"insights-operator-7c4986d449-nzxzl\" (UID: \"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8\") " pod="openshift-insights/insights-operator-7c4986d449-nzxzl" Oct 30 00:04:34.216705 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214785 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/526cc561-1bbd-4079-8a2f-326f939753e0-ca-trust-extracted\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:34.217397 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.216806 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/526cc561-1bbd-4079-8a2f-326f939753e0-registry-certificates\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:34.217397 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.217197 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/526cc561-1bbd-4079-8a2f-326f939753e0-image-registry-private-configuration\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:34.217397 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.217291 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 00:04:34.217397 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.217298 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0d3be8a7-773b-42e1-a9a1-60d7df717301-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7m2x2\" (UID: \"0d3be8a7-773b-42e1-a9a1-60d7df717301\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7m2x2" Oct 30 00:04:34.217397 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.217331 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert podName:0d3be8a7-773b-42e1-a9a1-60d7df717301 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:34.717318884 +0000 UTC m=+31.660011718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7m2x2" (UID: "0d3be8a7-773b-42e1-a9a1-60d7df717301") : secret "networking-console-plugin-cert" not found Oct 30 00:04:34.217397 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.217350 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d080b8e-cca6-4040-830e-cc9e9e25c50a-service-ca-bundle podName:9d080b8e-cca6-4040-830e-cc9e9e25c50a nodeName:}" failed. No retries permitted until 2026-10-30 00:04:34.717340932 +0000 UTC m=+31.660033763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9d080b8e-cca6-4040-830e-cc9e9e25c50a-service-ca-bundle") pod "router-default-645c8fd54c-hp29d" (UID: "9d080b8e-cca6-4040-830e-cc9e9e25c50a") : configmap references non-existent config key: service-ca.crt Oct 30 00:04:34.217753 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.217438 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-l25ts\" (UID: \"18a4ebbe-c390-49d8-99e7-858bfaa3b834\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l25ts" Oct 30 00:04:34.217753 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.217671 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-tmp\") pod \"insights-operator-7c4986d449-nzxzl\" (UID: \"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8\") " pod="openshift-insights/insights-operator-7c4986d449-nzxzl" Oct 30 00:04:34.218235 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.214785 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-snapshots\") pod \"insights-operator-7c4986d449-nzxzl\" (UID: \"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8\") " pod="openshift-insights/insights-operator-7c4986d449-nzxzl" Oct 30 00:04:34.218349 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.218260 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-serving-cert\") pod \"insights-operator-7c4986d449-nzxzl\" (UID: \"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8\") " pod="openshift-insights/insights-operator-7c4986d449-nzxzl" Oct 30 00:04:34.218349 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.218296 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1a647f1-e715-4a8a-8b44-0196b3b1af27-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l\" (UID: \"c1a647f1-e715-4a8a-8b44-0196b3b1af27\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l" Oct 30 00:04:34.218795 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.218776 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-trusted-ca\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:34.219042 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.219022 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9fa688d8-29ca-4c1f-926a-f1a4099e6b44-serving-cert\") pod \"console-operator-7cd56b8dbc-z8gb5\" (UID: \"9fa688d8-29ca-4c1f-926a-f1a4099e6b44\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" Oct 30 00:04:34.219241 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.219220 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a82abd0-5057-4361-9b0d-7a4f23ae97d1-serving-cert\") pod \"service-ca-operator-544d749c8c-6d24p\" (UID: \"8a82abd0-5057-4361-9b0d-7a4f23ae97d1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6d24p" Oct 30 00:04:34.219745 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.219722 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1a647f1-e715-4a8a-8b44-0196b3b1af27-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l\" (UID: \"c1a647f1-e715-4a8a-8b44-0196b3b1af27\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l" Oct 30 00:04:34.219840 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.219742 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-default-certificate\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:34.219840 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.219822 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:04:34.219840 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.219837 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dcddbb874-9zx5v: secret "image-registry-tls" not found Oct 30 00:04:34.220005 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.219888 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-registry-tls podName:526cc561-1bbd-4079-8a2f-326f939753e0 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:34.71987489 +0000 UTC m=+31.662567723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-registry-tls") pod "image-registry-7dcddbb874-9zx5v" (UID: "526cc561-1bbd-4079-8a2f-326f939753e0") : secret "image-registry-tls" not found Oct 30 00:04:34.220122 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.220101 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-image-registry-private-configuration\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:34.220183 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.220145 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-stats-auth\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:34.221022 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.220643 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-nzxzl\" (UID: \"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8\") " pod="openshift-insights/insights-operator-7c4986d449-nzxzl" Oct 30 00:04:34.221022 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.220842 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/acc82f37-fad2-424d-89d0-5e5876cb75de-config-volume\") pod \"dns-default-52kw4\" (UID: \"acc82f37-fad2-424d-89d0-5e5876cb75de\") " pod="openshift-dns/dns-default-52kw4" Oct 30 00:04:34.221022 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.220860 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/526cc561-1bbd-4079-8a2f-326f939753e0-trusted-ca\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:34.221022 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.220863 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de19ca9c-b22d-49a4-a5a6-a5f3a61bf892-cert\") pod \"ingress-canary-mpg4t\" (UID: \"de19ca9c-b22d-49a4-a5a6-a5f3a61bf892\") " pod="openshift-ingress-canary/ingress-canary-mpg4t" Oct 30 00:04:34.221022 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.220985 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-ca-trust-extracted\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:34.221303 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.221110 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/aff3a44b-6cad-449e-8733-f85e9899ecc3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-nqw98\" (UID: \"aff3a44b-6cad-449e-8733-f85e9899ecc3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nqw98" Oct 30 00:04:34.221303 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.221123 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9fa688d8-29ca-4c1f-926a-f1a4099e6b44-trusted-ca\") pod \"console-operator-7cd56b8dbc-z8gb5\" (UID: \"9fa688d8-29ca-4c1f-926a-f1a4099e6b44\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" Oct 30 00:04:34.221561 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.221540 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-installation-pull-secrets\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:34.222125 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.222106 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/526cc561-1bbd-4079-8a2f-326f939753e0-installation-pull-secrets\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:34.239681 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.239656 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ht68\" (UniqueName: \"kubernetes.io/projected/9d080b8e-cca6-4040-830e-cc9e9e25c50a-kube-api-access-6ht68\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:34.240523 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.240504 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-m98x7" Oct 30 00:04:34.240830 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.240788 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-bound-sa-token\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:34.240830 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.240823 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd5wb\" (UniqueName: \"kubernetes.io/projected/8a82abd0-5057-4361-9b0d-7a4f23ae97d1-kube-api-access-xd5wb\") pod \"service-ca-operator-544d749c8c-6d24p\" (UID: \"8a82abd0-5057-4361-9b0d-7a4f23ae97d1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6d24p" Oct 30 00:04:34.242036 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.242014 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v899\" (UniqueName: \"kubernetes.io/projected/c1a647f1-e715-4a8a-8b44-0196b3b1af27-kube-api-access-2v899\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l\" (UID: \"c1a647f1-e715-4a8a-8b44-0196b3b1af27\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l" Oct 30 00:04:34.242172 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.242146 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-bound-sa-token\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:34.243288 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.243266 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfc4x\" (UniqueName: \"kubernetes.io/projected/acc82f37-fad2-424d-89d0-5e5876cb75de-kube-api-access-kfc4x\") pod \"dns-default-52kw4\" (UID: \"acc82f37-fad2-424d-89d0-5e5876cb75de\") " pod="openshift-dns/dns-default-52kw4" Oct 30 00:04:34.243594 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.243574 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cngbs\" (UniqueName: \"kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-kube-api-access-cngbs\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:34.243661 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.243600 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cv5p\" (UniqueName: \"kubernetes.io/projected/9fa688d8-29ca-4c1f-926a-f1a4099e6b44-kube-api-access-6cv5p\") pod \"console-operator-7cd56b8dbc-z8gb5\" (UID: \"9fa688d8-29ca-4c1f-926a-f1a4099e6b44\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" Oct 30 00:04:34.243801 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.243774 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jzxf\" (UniqueName: \"kubernetes.io/projected/4350294b-64f3-4efd-9619-90c013d82432-kube-api-access-7jzxf\") pod \"volume-data-source-validator-58c9f5f46d-n78wz\" (UID: \"4350294b-64f3-4efd-9619-90c013d82432\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-n78wz" Oct 30 00:04:34.244308 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.244289 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mglpv\" (UniqueName: \"kubernetes.io/projected/de19ca9c-b22d-49a4-a5a6-a5f3a61bf892-kube-api-access-mglpv\") pod \"ingress-canary-mpg4t\" (UID: \"de19ca9c-b22d-49a4-a5a6-a5f3a61bf892\") " pod="openshift-ingress-canary/ingress-canary-mpg4t" Oct 30 00:04:34.248323 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.248304 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f87wl\" (UniqueName: \"kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-kube-api-access-f87wl\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:34.248544 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.248531 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-b9fcp" Oct 30 00:04:34.248895 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.248875 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8xpk\" (UniqueName: \"kubernetes.io/projected/18a4ebbe-c390-49d8-99e7-858bfaa3b834-kube-api-access-s8xpk\") pod \"cluster-samples-operator-6ddb6c8b94-l25ts\" (UID: \"18a4ebbe-c390-49d8-99e7-858bfaa3b834\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l25ts" Oct 30 00:04:34.249116 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.249101 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xrgq\" (UniqueName: \"kubernetes.io/projected/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-kube-api-access-6xrgq\") pod \"insights-operator-7c4986d449-nzxzl\" (UID: \"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8\") " pod="openshift-insights/insights-operator-7c4986d449-nzxzl" Oct 30 00:04:34.254554 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.254536 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-4fpvt" Oct 30 00:04:34.255035 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.255015 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56j8l\" (UniqueName: \"kubernetes.io/projected/aff3a44b-6cad-449e-8733-f85e9899ecc3-kube-api-access-56j8l\") pod \"cluster-monitoring-operator-884576c57-nqw98\" (UID: \"aff3a44b-6cad-449e-8733-f85e9899ecc3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nqw98" Oct 30 00:04:34.255777 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.255757 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dpbj\" (UniqueName: \"kubernetes.io/projected/32309642-7cba-4361-83b3-4680659c0544-kube-api-access-5dpbj\") pod \"network-check-source-6f78d59f99-glf29\" (UID: \"32309642-7cba-4361-83b3-4680659c0544\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-glf29" Oct 30 00:04:34.272586 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.272565 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" Oct 30 00:04:34.291489 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.291457 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-nzxzl" Oct 30 00:04:34.318396 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.318367 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-n78wz" Oct 30 00:04:34.323612 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.323586 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-glf29" Oct 30 00:04:34.342271 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.342246 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6d24p" Oct 30 00:04:34.348257 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.347632 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l" Oct 30 00:04:34.360413 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.360386 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mpg4t" Oct 30 00:04:34.592962 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.592936 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxvqx" Oct 30 00:04:34.608835 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.608815 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-pxfs7\"" Oct 30 00:04:34.639705 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.639680 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-b9fcp"] Oct 30 00:04:34.643263 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.643243 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-m98x7"] Oct 30 00:04:34.644053 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:34.644028 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f2b3396_a95d_4bc7_902f_1daf3dd689fd.slice/crio-cae5f4674cc009c6bbbe513a11b55d9f505be0f44cf15567f37a9eaec203b977 WatchSource:0}: Error finding container cae5f4674cc009c6bbbe513a11b55d9f505be0f44cf15567f37a9eaec203b977: Status 404 returned error can't find the container with id cae5f4674cc009c6bbbe513a11b55d9f505be0f44cf15567f37a9eaec203b977 Oct 30 00:04:34.644329 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.644307 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4fpvt"] Oct 30 00:04:34.644499 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:34.644475 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ce5e93_7d27_4fa6_b360_e571d8f18481.slice/crio-062d2ebaf412c97f991de7bda2e31d48a84793a3b68a7df39804ea228970a3cb WatchSource:0}: Error finding container 062d2ebaf412c97f991de7bda2e31d48a84793a3b68a7df39804ea228970a3cb: Status 404 returned error can't find the container with id 062d2ebaf412c97f991de7bda2e31d48a84793a3b68a7df39804ea228970a3cb Oct 30 00:04:34.645382 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:34.645349 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2413398_197b_4cad_8dc4_9bba9effe3b7.slice/crio-bbcafe6f8a6b21a56e4df1a6215e41026812e4fc5f2e8a3a816e0940661f3f49 WatchSource:0}: Error finding container bbcafe6f8a6b21a56e4df1a6215e41026812e4fc5f2e8a3a816e0940661f3f49: Status 404 returned error can't find the container with id bbcafe6f8a6b21a56e4df1a6215e41026812e4fc5f2e8a3a816e0940661f3f49 Oct 30 00:04:34.662390 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.662370 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-nzxzl"] Oct 30 00:04:34.665190 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:34.665167 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33f61ea8_5b1a_405a_aa36_7f07d79e2ec8.slice/crio-2de8ec0b28a0011e709a226b0a443eb6a46110a90d420e78df8898b980728aed WatchSource:0}: Error finding container 2de8ec0b28a0011e709a226b0a443eb6a46110a90d420e78df8898b980728aed: Status 404 returned error can't find the container with id 2de8ec0b28a0011e709a226b0a443eb6a46110a90d420e78df8898b980728aed Oct 30 00:04:34.708723 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.708693 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-z8gb5"] Oct 30 00:04:34.716699 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:34.716664 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fa688d8_29ca_4c1f_926a_f1a4099e6b44.slice/crio-45400f99a6a084f4caa35e5d1fa8e58afc39f1b891852ef408faf555bef06b64 WatchSource:0}: Error finding container 45400f99a6a084f4caa35e5d1fa8e58afc39f1b891852ef408faf555bef06b64: Status 404 returned error can't find the container with id 45400f99a6a084f4caa35e5d1fa8e58afc39f1b891852ef408faf555bef06b64 Oct 30 00:04:34.721384 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.721355 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d080b8e-cca6-4040-830e-cc9e9e25c50a-service-ca-bundle\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:34.721495 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.721412 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-metrics-certs\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:34.721495 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.721448 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-registry-tls\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:34.721587 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.721503 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acc82f37-fad2-424d-89d0-5e5876cb75de-metrics-tls\") pod \"dns-default-52kw4\" (UID: \"acc82f37-fad2-424d-89d0-5e5876cb75de\") " pod="openshift-dns/dns-default-52kw4" Oct 30 00:04:34.721587 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.721563 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7m2x2\" (UID: \"0d3be8a7-773b-42e1-a9a1-60d7df717301\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7m2x2" Oct 30 00:04:34.721684 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.721596 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l25ts\" (UID: \"18a4ebbe-c390-49d8-99e7-858bfaa3b834\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l25ts" Oct 30 00:04:34.721684 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.721618 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-tls\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:34.721684 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.721660 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aff3a44b-6cad-449e-8733-f85e9899ecc3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nqw98\" (UID: \"aff3a44b-6cad-449e-8733-f85e9899ecc3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nqw98" Oct 30 00:04:34.721829 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.721813 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 00:04:34.721875 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.721867 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff3a44b-6cad-449e-8733-f85e9899ecc3-cluster-monitoring-operator-tls podName:aff3a44b-6cad-449e-8733-f85e9899ecc3 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:35.72184993 +0000 UTC m=+32.664542757 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/aff3a44b-6cad-449e-8733-f85e9899ecc3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nqw98" (UID: "aff3a44b-6cad-449e-8733-f85e9899ecc3") : secret "cluster-monitoring-operator-tls" not found Oct 30 00:04:34.721964 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.721951 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d080b8e-cca6-4040-830e-cc9e9e25c50a-service-ca-bundle podName:9d080b8e-cca6-4040-830e-cc9e9e25c50a nodeName:}" failed. No retries permitted until 2026-10-30 00:04:35.721940478 +0000 UTC m=+32.664633312 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9d080b8e-cca6-4040-830e-cc9e9e25c50a-service-ca-bundle") pod "router-default-645c8fd54c-hp29d" (UID: "9d080b8e-cca6-4040-830e-cc9e9e25c50a") : configmap references non-existent config key: service-ca.crt Oct 30 00:04:34.722036 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.722014 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 00:04:34.722084 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.722045 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-metrics-certs podName:9d080b8e-cca6-4040-830e-cc9e9e25c50a nodeName:}" failed. No retries permitted until 2026-10-30 00:04:35.722034512 +0000 UTC m=+32.664727343 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-metrics-certs") pod "router-default-645c8fd54c-hp29d" (UID: "9d080b8e-cca6-4040-830e-cc9e9e25c50a") : secret "router-metrics-certs-default" not found Oct 30 00:04:34.722136 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.722103 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:04:34.722136 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.722113 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dcddbb874-9zx5v: secret "image-registry-tls" not found Oct 30 00:04:34.722228 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.722143 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-registry-tls podName:526cc561-1bbd-4079-8a2f-326f939753e0 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:35.722133334 +0000 UTC m=+32.664826172 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-registry-tls") pod "image-registry-7dcddbb874-9zx5v" (UID: "526cc561-1bbd-4079-8a2f-326f939753e0") : secret "image-registry-tls" not found Oct 30 00:04:34.722228 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.722195 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:04:34.722228 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.722223 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acc82f37-fad2-424d-89d0-5e5876cb75de-metrics-tls podName:acc82f37-fad2-424d-89d0-5e5876cb75de nodeName:}" failed. No retries permitted until 2026-10-30 00:04:35.722213971 +0000 UTC m=+32.664906791 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acc82f37-fad2-424d-89d0-5e5876cb75de-metrics-tls") pod "dns-default-52kw4" (UID: "acc82f37-fad2-424d-89d0-5e5876cb75de") : secret "dns-default-metrics-tls" not found Oct 30 00:04:34.722431 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.722276 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 00:04:34.722431 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.722306 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert podName:0d3be8a7-773b-42e1-a9a1-60d7df717301 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:35.722296701 +0000 UTC m=+32.664989524 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7m2x2" (UID: "0d3be8a7-773b-42e1-a9a1-60d7df717301") : secret "networking-console-plugin-cert" not found Oct 30 00:04:34.722431 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.722358 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 00:04:34.722431 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.722386 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-tls podName:18a4ebbe-c390-49d8-99e7-858bfaa3b834 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:35.722377132 +0000 UTC m=+32.665069965 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-l25ts" (UID: "18a4ebbe-c390-49d8-99e7-858bfaa3b834") : secret "samples-operator-tls" not found Oct 30 00:04:34.722636 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.722443 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:04:34.722636 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.722453 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c8c8757f-tspqd: secret "image-registry-tls" not found Oct 30 00:04:34.722636 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:34.722482 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-tls podName:fd8f9a4a-5b77-4a52-9ea9-a12245c00489 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:35.722472409 +0000 UTC m=+32.665165230 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-tls") pod "image-registry-7c8c8757f-tspqd" (UID: "fd8f9a4a-5b77-4a52-9ea9-a12245c00489") : secret "image-registry-tls" not found Oct 30 00:04:34.739932 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.733171 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mpg4t"] Oct 30 00:04:34.744345 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:34.744318 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde19ca9c_b22d_49a4_a5a6_a5f3a61bf892.slice/crio-8710df781396f3314540cd1fdb27dfde39ee527d286c2e27699590bb90d4588d WatchSource:0}: Error finding container 8710df781396f3314540cd1fdb27dfde39ee527d286c2e27699590bb90d4588d: Status 404 returned error can't find the container with id 8710df781396f3314540cd1fdb27dfde39ee527d286c2e27699590bb90d4588d Oct 30 00:04:34.812670 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.812639 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-n78wz"] Oct 30 00:04:34.813390 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.813283 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-nzxzl" event={"ID":"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8","Type":"ContainerStarted","Data":"2de8ec0b28a0011e709a226b0a443eb6a46110a90d420e78df8898b980728aed"} Oct 30 00:04:34.814742 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.814720 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-6d24p"] Oct 30 00:04:34.814882 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.814832 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mpg4t" event={"ID":"de19ca9c-b22d-49a4-a5a6-a5f3a61bf892","Type":"ContainerStarted","Data":"8710df781396f3314540cd1fdb27dfde39ee527d286c2e27699590bb90d4588d"} Oct 30 00:04:34.816644 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.816290 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-glf29"] Oct 30 00:04:34.816644 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.816543 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-b9fcp" event={"ID":"6f2b3396-a95d-4bc7-902f-1daf3dd689fd","Type":"ContainerStarted","Data":"cae5f4674cc009c6bbbe513a11b55d9f505be0f44cf15567f37a9eaec203b977"} Oct 30 00:04:34.817706 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:34.817679 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4350294b_64f3_4efd_9619_90c013d82432.slice/crio-2a928774f503438c73c71ebc467269f535dce7bfd334c709327ea46cdfd13a9b WatchSource:0}: Error finding container 2a928774f503438c73c71ebc467269f535dce7bfd334c709327ea46cdfd13a9b: Status 404 returned error can't find the container with id 2a928774f503438c73c71ebc467269f535dce7bfd334c709327ea46cdfd13a9b Oct 30 00:04:34.817957 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.817901 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" event={"ID":"9fa688d8-29ca-4c1f-926a-f1a4099e6b44","Type":"ContainerStarted","Data":"45400f99a6a084f4caa35e5d1fa8e58afc39f1b891852ef408faf555bef06b64"} Oct 30 00:04:34.818566 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:34.818526 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32309642_7cba_4361_83b3_4680659c0544.slice/crio-938033fc66565d9fc00b418f0e9cf97a5d76e65f9284a69ab3cd167e0a482912 WatchSource:0}: Error finding container 938033fc66565d9fc00b418f0e9cf97a5d76e65f9284a69ab3cd167e0a482912: Status 404 returned error can't find the container with id 938033fc66565d9fc00b418f0e9cf97a5d76e65f9284a69ab3cd167e0a482912 Oct 30 00:04:34.819270 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:34.819247 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a82abd0_5057_4361_9b0d_7a4f23ae97d1.slice/crio-7347e4b08319778fdcd85e69581d07afc3de3b53fc4f15668b00d27490ee6793 WatchSource:0}: Error finding container 7347e4b08319778fdcd85e69581d07afc3de3b53fc4f15668b00d27490ee6793: Status 404 returned error can't find the container with id 7347e4b08319778fdcd85e69581d07afc3de3b53fc4f15668b00d27490ee6793 Oct 30 00:04:34.819654 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.819632 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-4fpvt" event={"ID":"c3ce5e93-7d27-4fa6-b360-e571d8f18481","Type":"ContainerStarted","Data":"70c06aff96a744fd8eb7edee5c62f0a3242f591f7bb757998065c4be1ddad065"} Oct 30 00:04:34.819710 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.819668 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-4fpvt" event={"ID":"c3ce5e93-7d27-4fa6-b360-e571d8f18481","Type":"ContainerStarted","Data":"062d2ebaf412c97f991de7bda2e31d48a84793a3b68a7df39804ea228970a3cb"} Oct 30 00:04:34.821327 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.821217 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-m98x7" event={"ID":"d2413398-197b-4cad-8dc4-9bba9effe3b7","Type":"ContainerStarted","Data":"bbcafe6f8a6b21a56e4df1a6215e41026812e4fc5f2e8a3a816e0940661f3f49"} Oct 30 00:04:34.853338 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:34.853298 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l"] Oct 30 00:04:34.856004 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:34.855977 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a647f1_e715_4a8a_8b44_0196b3b1af27.slice/crio-1f4f8d7cfd7163a3446b5059aa1b4fa1ee1529c9f2ad7703868530c29b5acbae WatchSource:0}: Error finding container 1f4f8d7cfd7163a3446b5059aa1b4fa1ee1529c9f2ad7703868530c29b5acbae: Status 404 returned error can't find the container with id 1f4f8d7cfd7163a3446b5059aa1b4fa1ee1529c9f2ad7703868530c29b5acbae Oct 30 00:04:35.591828 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:35.591794 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:35.592305 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:35.592289 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:35.608533 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:35.608468 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-thx59\"" Oct 30 00:04:35.609492 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:35.608971 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 00:04:35.609492 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:35.609076 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-m7wf2\"" Oct 30 00:04:35.609492 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:35.609145 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 00:04:35.733426 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:35.731817 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aff3a44b-6cad-449e-8733-f85e9899ecc3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nqw98\" (UID: \"aff3a44b-6cad-449e-8733-f85e9899ecc3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nqw98" Oct 30 00:04:35.733426 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:35.731897 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d080b8e-cca6-4040-830e-cc9e9e25c50a-service-ca-bundle\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:35.733426 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:35.731954 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-metrics-certs\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:35.733426 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:35.731987 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-registry-tls\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:35.733426 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:35.732039 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acc82f37-fad2-424d-89d0-5e5876cb75de-metrics-tls\") pod \"dns-default-52kw4\" (UID: \"acc82f37-fad2-424d-89d0-5e5876cb75de\") " pod="openshift-dns/dns-default-52kw4" Oct 30 00:04:35.733426 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:35.732101 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7m2x2\" (UID: \"0d3be8a7-773b-42e1-a9a1-60d7df717301\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7m2x2" Oct 30 00:04:35.733426 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:35.732136 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l25ts\" (UID: \"18a4ebbe-c390-49d8-99e7-858bfaa3b834\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l25ts" Oct 30 00:04:35.733426 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:35.732161 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-tls\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:35.733426 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:35.732297 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:04:35.733426 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:35.732320 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c8c8757f-tspqd: secret "image-registry-tls" not found Oct 30 00:04:35.733426 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:35.732375 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-tls podName:fd8f9a4a-5b77-4a52-9ea9-a12245c00489 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:37.732357512 +0000 UTC m=+34.675050336 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-tls") pod "image-registry-7c8c8757f-tspqd" (UID: "fd8f9a4a-5b77-4a52-9ea9-a12245c00489") : secret "image-registry-tls" not found Oct 30 00:04:35.733426 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:35.732795 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 00:04:35.733426 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:35.732844 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff3a44b-6cad-449e-8733-f85e9899ecc3-cluster-monitoring-operator-tls podName:aff3a44b-6cad-449e-8733-f85e9899ecc3 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:37.732829028 +0000 UTC m=+34.675521889 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/aff3a44b-6cad-449e-8733-f85e9899ecc3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nqw98" (UID: "aff3a44b-6cad-449e-8733-f85e9899ecc3") : secret "cluster-monitoring-operator-tls" not found Oct 30 00:04:35.733426 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:35.732929 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d080b8e-cca6-4040-830e-cc9e9e25c50a-service-ca-bundle podName:9d080b8e-cca6-4040-830e-cc9e9e25c50a nodeName:}" failed. No retries permitted until 2026-10-30 00:04:37.732903091 +0000 UTC m=+34.675595916 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9d080b8e-cca6-4040-830e-cc9e9e25c50a-service-ca-bundle") pod "router-default-645c8fd54c-hp29d" (UID: "9d080b8e-cca6-4040-830e-cc9e9e25c50a") : configmap references non-existent config key: service-ca.crt Oct 30 00:04:35.733426 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:35.732984 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 00:04:35.734263 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:35.733012 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-metrics-certs podName:9d080b8e-cca6-4040-830e-cc9e9e25c50a nodeName:}" failed. No retries permitted until 2026-10-30 00:04:37.733003594 +0000 UTC m=+34.675696419 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-metrics-certs") pod "router-default-645c8fd54c-hp29d" (UID: "9d080b8e-cca6-4040-830e-cc9e9e25c50a") : secret "router-metrics-certs-default" not found Oct 30 00:04:35.734263 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:35.733071 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:04:35.734263 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:35.733082 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dcddbb874-9zx5v: secret "image-registry-tls" not found Oct 30 00:04:35.734263 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:35.733111 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-registry-tls podName:526cc561-1bbd-4079-8a2f-326f939753e0 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:37.73310153 +0000 UTC m=+34.675794352 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-registry-tls") pod "image-registry-7dcddbb874-9zx5v" (UID: "526cc561-1bbd-4079-8a2f-326f939753e0") : secret "image-registry-tls" not found Oct 30 00:04:35.734263 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:35.733165 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:04:35.734263 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:35.733211 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acc82f37-fad2-424d-89d0-5e5876cb75de-metrics-tls podName:acc82f37-fad2-424d-89d0-5e5876cb75de nodeName:}" failed. No retries permitted until 2026-10-30 00:04:37.733182491 +0000 UTC m=+34.675875311 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acc82f37-fad2-424d-89d0-5e5876cb75de-metrics-tls") pod "dns-default-52kw4" (UID: "acc82f37-fad2-424d-89d0-5e5876cb75de") : secret "dns-default-metrics-tls" not found Oct 30 00:04:35.734263 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:35.733269 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 00:04:35.734263 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:35.733297 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert podName:0d3be8a7-773b-42e1-a9a1-60d7df717301 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:37.733287772 +0000 UTC m=+34.675980593 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7m2x2" (UID: "0d3be8a7-773b-42e1-a9a1-60d7df717301") : secret "networking-console-plugin-cert" not found Oct 30 00:04:35.734263 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:35.733353 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 00:04:35.734263 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:35.733381 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-tls podName:18a4ebbe-c390-49d8-99e7-858bfaa3b834 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:37.733371864 +0000 UTC m=+34.676064687 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-l25ts" (UID: "18a4ebbe-c390-49d8-99e7-858bfaa3b834") : secret "samples-operator-tls" not found Oct 30 00:04:35.832089 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:35.832054 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-b9fcp" event={"ID":"6f2b3396-a95d-4bc7-902f-1daf3dd689fd","Type":"ContainerStarted","Data":"219a37e0a1360eaab39257014023c9ec6f648e83be057e4f0270135d798e76fb"} Oct 30 00:04:35.834105 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:35.834052 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6d24p" event={"ID":"8a82abd0-5057-4361-9b0d-7a4f23ae97d1","Type":"ContainerStarted","Data":"7347e4b08319778fdcd85e69581d07afc3de3b53fc4f15668b00d27490ee6793"} Oct 30 00:04:35.836252 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:35.836223 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l" event={"ID":"c1a647f1-e715-4a8a-8b44-0196b3b1af27","Type":"ContainerStarted","Data":"1f4f8d7cfd7163a3446b5059aa1b4fa1ee1529c9f2ad7703868530c29b5acbae"} Oct 30 00:04:35.838923 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:35.838889 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-glf29" event={"ID":"32309642-7cba-4361-83b3-4680659c0544","Type":"ContainerStarted","Data":"938033fc66565d9fc00b418f0e9cf97a5d76e65f9284a69ab3cd167e0a482912"} Oct 30 00:04:35.841110 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:35.841059 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-m98x7" event={"ID":"d2413398-197b-4cad-8dc4-9bba9effe3b7","Type":"ContainerStarted","Data":"5692a5f938a4f1fb0ca768ab91ce6b4fe7dafd2a2bf9c7451b1ad92e3f8b99d5"} Oct 30 00:04:35.844086 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:35.843663 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-n78wz" event={"ID":"4350294b-64f3-4efd-9619-90c013d82432","Type":"ContainerStarted","Data":"2a928774f503438c73c71ebc467269f535dce7bfd334c709327ea46cdfd13a9b"} Oct 30 00:04:35.873776 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:35.872655 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-b9fcp" podStartSLOduration=246.872638477 podStartE2EDuration="4m6.872638477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:00:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:35.871893169 +0000 UTC m=+32.814586013" watchObservedRunningTime="2026-10-30 00:04:35.872638477 +0000 UTC m=+32.815331320" Oct 30 00:04:35.913986 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:35.912660 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-4fpvt" podStartSLOduration=246.912643817 podStartE2EDuration="4m6.912643817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:00:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:35.911549643 +0000 UTC m=+32.854242487" watchObservedRunningTime="2026-10-30 00:04:35.912643817 +0000 UTC m=+32.855336660" Oct 30 00:04:35.953024 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:35.952684 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-m98x7" podStartSLOduration=246.952664421 podStartE2EDuration="4m6.952664421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:00:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:35.951245292 +0000 UTC m=+32.893938147" watchObservedRunningTime="2026-10-30 00:04:35.952664421 +0000 UTC m=+32.895357277" Oct 30 00:04:37.348583 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:37.348535 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs\") pod \"network-metrics-daemon-52qpr\" (UID: \"30673bf9-d332-4592-afce-b16db49876cb\") " pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:37.349264 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:37.348861 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 00:04:37.349264 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:37.348938 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs podName:30673bf9-d332-4592-afce-b16db49876cb nodeName:}" failed. No retries permitted until 2026-10-30 00:05:09.348905722 +0000 UTC m=+66.291598555 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs") pod "network-metrics-daemon-52qpr" (UID: "30673bf9-d332-4592-afce-b16db49876cb") : secret "metrics-daemon-secret" not found Oct 30 00:04:37.451115 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:37.450096 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb8917db-c1ba-43af-995e-1948de69aa29-original-pull-secret\") pod \"global-pull-secret-syncer-zm28x\" (UID: \"fb8917db-c1ba-43af-995e-1948de69aa29\") " pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:37.451115 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:37.450200 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpg2j\" (UniqueName: \"kubernetes.io/projected/a20efc24-e4cb-4075-911e-bfc43dcfc20b-kube-api-access-dpg2j\") pod \"network-check-target-sxvqx\" (UID: \"a20efc24-e4cb-4075-911e-bfc43dcfc20b\") " pod="openshift-network-diagnostics/network-check-target-sxvqx" Oct 30 00:04:37.460478 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:37.460410 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpg2j\" (UniqueName: \"kubernetes.io/projected/a20efc24-e4cb-4075-911e-bfc43dcfc20b-kube-api-access-dpg2j\") pod \"network-check-target-sxvqx\" (UID: \"a20efc24-e4cb-4075-911e-bfc43dcfc20b\") " pod="openshift-network-diagnostics/network-check-target-sxvqx" Oct 30 00:04:37.464356 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:37.464252 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb8917db-c1ba-43af-995e-1948de69aa29-original-pull-secret\") pod \"global-pull-secret-syncer-zm28x\" (UID: \"fb8917db-c1ba-43af-995e-1948de69aa29\") " pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:37.640093 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:37.640014 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxvqx" Oct 30 00:04:37.734806 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:37.734227 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:37.754027 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:37.753475 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-metrics-certs\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:37.754027 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:37.753511 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-registry-tls\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:37.754027 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:37.753547 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acc82f37-fad2-424d-89d0-5e5876cb75de-metrics-tls\") pod \"dns-default-52kw4\" (UID: \"acc82f37-fad2-424d-89d0-5e5876cb75de\") " pod="openshift-dns/dns-default-52kw4" Oct 30 00:04:37.754027 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:37.753579 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7m2x2\" (UID: \"0d3be8a7-773b-42e1-a9a1-60d7df717301\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7m2x2" Oct 30 00:04:37.754027 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:37.753601 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l25ts\" (UID: \"18a4ebbe-c390-49d8-99e7-858bfaa3b834\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l25ts" Oct 30 00:04:37.754027 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:37.753619 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-tls\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:37.754027 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:37.753641 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aff3a44b-6cad-449e-8733-f85e9899ecc3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nqw98\" (UID: \"aff3a44b-6cad-449e-8733-f85e9899ecc3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nqw98" Oct 30 00:04:37.754027 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:37.753671 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d080b8e-cca6-4040-830e-cc9e9e25c50a-service-ca-bundle\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:37.755008 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:37.754149 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d080b8e-cca6-4040-830e-cc9e9e25c50a-service-ca-bundle podName:9d080b8e-cca6-4040-830e-cc9e9e25c50a nodeName:}" failed. No retries permitted until 2026-10-30 00:04:41.754128341 +0000 UTC m=+38.696821175 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9d080b8e-cca6-4040-830e-cc9e9e25c50a-service-ca-bundle") pod "router-default-645c8fd54c-hp29d" (UID: "9d080b8e-cca6-4040-830e-cc9e9e25c50a") : configmap references non-existent config key: service-ca.crt Oct 30 00:04:37.755008 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:37.754608 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 00:04:37.755008 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:37.754632 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 00:04:37.755008 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:37.754666 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-metrics-certs podName:9d080b8e-cca6-4040-830e-cc9e9e25c50a nodeName:}" failed. No retries permitted until 2026-10-30 00:04:41.754649261 +0000 UTC m=+38.697342101 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-metrics-certs") pod "router-default-645c8fd54c-hp29d" (UID: "9d080b8e-cca6-4040-830e-cc9e9e25c50a") : secret "router-metrics-certs-default" not found Oct 30 00:04:37.755008 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:37.754683 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-tls podName:18a4ebbe-c390-49d8-99e7-858bfaa3b834 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:41.754673976 +0000 UTC m=+38.697366796 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-l25ts" (UID: "18a4ebbe-c390-49d8-99e7-858bfaa3b834") : secret "samples-operator-tls" not found Oct 30 00:04:37.755008 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:37.754710 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:04:37.755008 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:37.754721 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dcddbb874-9zx5v: secret "image-registry-tls" not found Oct 30 00:04:37.755008 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:37.754740 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:04:37.755008 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:37.754751 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-registry-tls podName:526cc561-1bbd-4079-8a2f-326f939753e0 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:41.754739858 +0000 UTC m=+38.697432690 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-registry-tls") pod "image-registry-7dcddbb874-9zx5v" (UID: "526cc561-1bbd-4079-8a2f-326f939753e0") : secret "image-registry-tls" not found Oct 30 00:04:37.755008 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:37.754751 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c8c8757f-tspqd: secret "image-registry-tls" not found Oct 30 00:04:37.755008 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:37.754783 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-tls podName:fd8f9a4a-5b77-4a52-9ea9-a12245c00489 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:41.754774411 +0000 UTC m=+38.697467244 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-tls") pod "image-registry-7c8c8757f-tspqd" (UID: "fd8f9a4a-5b77-4a52-9ea9-a12245c00489") : secret "image-registry-tls" not found Oct 30 00:04:37.755008 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:37.754801 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 00:04:37.755008 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:37.754832 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff3a44b-6cad-449e-8733-f85e9899ecc3-cluster-monitoring-operator-tls podName:aff3a44b-6cad-449e-8733-f85e9899ecc3 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:41.754823225 +0000 UTC m=+38.697516062 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/aff3a44b-6cad-449e-8733-f85e9899ecc3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nqw98" (UID: "aff3a44b-6cad-449e-8733-f85e9899ecc3") : secret "cluster-monitoring-operator-tls" not found Oct 30 00:04:37.755008 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:37.754833 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:04:37.755008 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:37.754861 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acc82f37-fad2-424d-89d0-5e5876cb75de-metrics-tls podName:acc82f37-fad2-424d-89d0-5e5876cb75de nodeName:}" failed. No retries permitted until 2026-10-30 00:04:41.754852875 +0000 UTC m=+38.697545709 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acc82f37-fad2-424d-89d0-5e5876cb75de-metrics-tls") pod "dns-default-52kw4" (UID: "acc82f37-fad2-424d-89d0-5e5876cb75de") : secret "dns-default-metrics-tls" not found Oct 30 00:04:37.755008 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:37.754877 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 00:04:37.755785 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:37.754923 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert podName:0d3be8a7-773b-42e1-a9a1-60d7df717301 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:41.754899489 +0000 UTC m=+38.697592322 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7m2x2" (UID: "0d3be8a7-773b-42e1-a9a1-60d7df717301") : secret "networking-console-plugin-cert" not found Oct 30 00:04:37.858030 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:37.857283 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-n78wz" event={"ID":"4350294b-64f3-4efd-9619-90c013d82432","Type":"ContainerStarted","Data":"f92f1b5477bf0e58706f55d96a5afe64a6c6ff67f9c2bee68ee0e32b454080c8"} Oct 30 00:04:37.899461 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:37.899265 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-n78wz" podStartSLOduration=218.710659668 podStartE2EDuration="3m40.899246248s" totalImagesPullingTime="2.18858658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:00:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:37.897600144 +0000 UTC m=+34.840292991" watchObservedRunningTime="2026-10-30 00:04:37.899246248 +0000 UTC m=+34.841939093" Oct 30 00:04:37.978627 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:37.978404 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sxvqx"] Oct 30 00:04:37.987456 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:37.987424 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda20efc24_e4cb_4075_911e_bfc43dcfc20b.slice/crio-8c48de3e101e60c8a453a3f8a53c5c73ef1f5df30b21f5c8f2d45e4752e501bc WatchSource:0}: Error finding container 8c48de3e101e60c8a453a3f8a53c5c73ef1f5df30b21f5c8f2d45e4752e501bc: Status 404 returned error can't find the container with id 8c48de3e101e60c8a453a3f8a53c5c73ef1f5df30b21f5c8f2d45e4752e501bc Oct 30 00:04:38.070838 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:38.070775 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zm28x"] Oct 30 00:04:38.861698 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:38.861667 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zm28x" event={"ID":"fb8917db-c1ba-43af-995e-1948de69aa29","Type":"ContainerStarted","Data":"f7a8bc416f053eedd7084a1813fbc0d84ac2599e5f8b7717bd3405aa0eba9a9c"} Oct 30 00:04:38.864354 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:38.864316 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sxvqx" event={"ID":"a20efc24-e4cb-4075-911e-bfc43dcfc20b","Type":"ContainerStarted","Data":"8c48de3e101e60c8a453a3f8a53c5c73ef1f5df30b21f5c8f2d45e4752e501bc"} Oct 30 00:04:39.875995 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:39.875289 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8gb5_9fa688d8-29ca-4c1f-926a-f1a4099e6b44/console-operator/0.log" Oct 30 00:04:39.875995 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:39.875335 2580 generic.go:356] "Generic (PLEG): container finished" podID="9fa688d8-29ca-4c1f-926a-f1a4099e6b44" containerID="dadbee92034e275ef20827808b6f9ee7bdfb4aa33030a53a09e30fd4c91474f6" exitCode=255 Oct 30 00:04:39.875995 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:39.875581 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" event={"ID":"9fa688d8-29ca-4c1f-926a-f1a4099e6b44","Type":"ContainerDied","Data":"dadbee92034e275ef20827808b6f9ee7bdfb4aa33030a53a09e30fd4c91474f6"} Oct 30 00:04:39.875995 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:39.875704 2580 scope.go:122] "RemoveContainer" containerID="dadbee92034e275ef20827808b6f9ee7bdfb4aa33030a53a09e30fd4c91474f6" Oct 30 00:04:39.879529 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:39.879468 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-nzxzl" event={"ID":"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8","Type":"ContainerStarted","Data":"06d35bd282ecc75a88f7686df991b0bfa14b3b765a5cf4ad69487217eed30751"} Oct 30 00:04:39.882008 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:39.881983 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l" event={"ID":"c1a647f1-e715-4a8a-8b44-0196b3b1af27","Type":"ContainerStarted","Data":"a4959dc8121b76faa5b758fa2cfcb5a2373e16e833f2613254ce12fad0ed987c"} Oct 30 00:04:40.015783 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.015725 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l" podStartSLOduration=218.24384712 podStartE2EDuration="3m43.015702268s" totalImagesPullingTime="4.771855148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:00:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:39.960223094 +0000 UTC m=+36.902915941" watchObservedRunningTime="2026-10-30 00:04:40.015702268 +0000 UTC m=+36.958395113" Oct 30 00:04:40.316146 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.316078 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-nzxzl" podStartSLOduration=211.99040969 podStartE2EDuration="3m36.316056669s" totalImagesPullingTime="4.325646979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:01:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:40.016888013 +0000 UTC m=+36.959580837" watchObservedRunningTime="2026-10-30 00:04:40.316056669 +0000 UTC m=+37.258749508" Oct 30 00:04:40.318038 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.318009 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dd8wx"] Oct 30 00:04:40.351134 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.341166 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dd8wx"] Oct 30 00:04:40.351134 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.341227 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:40.357143 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.357117 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bqlwc\"" Oct 30 00:04:40.357264 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.357168 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 00:04:40.357264 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.357193 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 00:04:40.409155 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.409125 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-nkc49-p6m54"] Oct 30 00:04:40.435478 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.435454 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-nkc49-p6m54"] Oct 30 00:04:40.435590 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.435572 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-nkc49-p6m54" Oct 30 00:04:40.459067 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.456963 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hslgr\"" Oct 30 00:04:40.485871 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.485836 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12aceb3a-2476-4efc-94d8-e6ec78cfc680-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:40.486036 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.485938 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/12aceb3a-2476-4efc-94d8-e6ec78cfc680-crio-socket\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:40.486036 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.485967 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12aceb3a-2476-4efc-94d8-e6ec78cfc680-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:40.486036 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.486017 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/12aceb3a-2476-4efc-94d8-e6ec78cfc680-data-volume\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:40.486209 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.486047 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vrjg\" (UniqueName: \"kubernetes.io/projected/12aceb3a-2476-4efc-94d8-e6ec78cfc680-kube-api-access-9vrjg\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:40.587274 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.587228 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/12aceb3a-2476-4efc-94d8-e6ec78cfc680-crio-socket\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:40.587274 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.587277 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12aceb3a-2476-4efc-94d8-e6ec78cfc680-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:40.587484 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.587316 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/12aceb3a-2476-4efc-94d8-e6ec78cfc680-data-volume\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:40.587484 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.587345 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09f5d7e2-51b8-4153-89c5-674d67065762-service-ca-bundle\") pod \"periodic-gathering-nkc49-p6m54\" (UID: \"09f5d7e2-51b8-4153-89c5-674d67065762\") " pod="openshift-insights/periodic-gathering-nkc49-p6m54" Oct 30 00:04:40.587484 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.587374 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vrjg\" (UniqueName: \"kubernetes.io/projected/12aceb3a-2476-4efc-94d8-e6ec78cfc680-kube-api-access-9vrjg\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:40.587572 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.587480 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/12aceb3a-2476-4efc-94d8-e6ec78cfc680-crio-socket\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:40.587753 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.587733 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/12aceb3a-2476-4efc-94d8-e6ec78cfc680-data-volume\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:40.587792 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.587487 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09f5d7e2-51b8-4153-89c5-674d67065762-trusted-ca-bundle\") pod \"periodic-gathering-nkc49-p6m54\" (UID: \"09f5d7e2-51b8-4153-89c5-674d67065762\") " pod="openshift-insights/periodic-gathering-nkc49-p6m54" Oct 30 00:04:40.587792 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.587787 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgkdb\" (UniqueName: \"kubernetes.io/projected/09f5d7e2-51b8-4153-89c5-674d67065762-kube-api-access-lgkdb\") pod \"periodic-gathering-nkc49-p6m54\" (UID: \"09f5d7e2-51b8-4153-89c5-674d67065762\") " pod="openshift-insights/periodic-gathering-nkc49-p6m54" Oct 30 00:04:40.587867 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:40.587814 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 00:04:40.587903 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:40.587869 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12aceb3a-2476-4efc-94d8-e6ec78cfc680-kube-rbac-proxy-cm podName:12aceb3a-2476-4efc-94d8-e6ec78cfc680 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:41.087850873 +0000 UTC m=+38.030543693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/12aceb3a-2476-4efc-94d8-e6ec78cfc680-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dd8wx" (UID: "12aceb3a-2476-4efc-94d8-e6ec78cfc680") : configmap "kube-rbac-proxy" not found Oct 30 00:04:40.587989 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.587967 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12aceb3a-2476-4efc-94d8-e6ec78cfc680-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:40.588024 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.588005 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/09f5d7e2-51b8-4153-89c5-674d67065762-archives-path\") pod \"periodic-gathering-nkc49-p6m54\" (UID: \"09f5d7e2-51b8-4153-89c5-674d67065762\") " pod="openshift-insights/periodic-gathering-nkc49-p6m54" Oct 30 00:04:40.588133 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:40.588111 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:04:40.588206 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:40.588169 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12aceb3a-2476-4efc-94d8-e6ec78cfc680-insights-runtime-extractor-tls podName:12aceb3a-2476-4efc-94d8-e6ec78cfc680 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:41.088154424 +0000 UTC m=+38.030847259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/12aceb3a-2476-4efc-94d8-e6ec78cfc680-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dd8wx" (UID: "12aceb3a-2476-4efc-94d8-e6ec78cfc680") : secret "insights-runtime-extractor-tls" not found Oct 30 00:04:40.612755 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.612696 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vrjg\" (UniqueName: \"kubernetes.io/projected/12aceb3a-2476-4efc-94d8-e6ec78cfc680-kube-api-access-9vrjg\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:40.689281 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.689252 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/09f5d7e2-51b8-4153-89c5-674d67065762-archives-path\") pod \"periodic-gathering-nkc49-p6m54\" (UID: \"09f5d7e2-51b8-4153-89c5-674d67065762\") " pod="openshift-insights/periodic-gathering-nkc49-p6m54" Oct 30 00:04:40.689384 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.689355 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09f5d7e2-51b8-4153-89c5-674d67065762-service-ca-bundle\") pod \"periodic-gathering-nkc49-p6m54\" (UID: \"09f5d7e2-51b8-4153-89c5-674d67065762\") " pod="openshift-insights/periodic-gathering-nkc49-p6m54" Oct 30 00:04:40.689451 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.689414 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09f5d7e2-51b8-4153-89c5-674d67065762-trusted-ca-bundle\") pod \"periodic-gathering-nkc49-p6m54\" (UID: \"09f5d7e2-51b8-4153-89c5-674d67065762\") " pod="openshift-insights/periodic-gathering-nkc49-p6m54" Oct 30 00:04:40.689500 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.689447 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgkdb\" (UniqueName: \"kubernetes.io/projected/09f5d7e2-51b8-4153-89c5-674d67065762-kube-api-access-lgkdb\") pod \"periodic-gathering-nkc49-p6m54\" (UID: \"09f5d7e2-51b8-4153-89c5-674d67065762\") " pod="openshift-insights/periodic-gathering-nkc49-p6m54" Oct 30 00:04:40.690216 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.690189 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09f5d7e2-51b8-4153-89c5-674d67065762-service-ca-bundle\") pod \"periodic-gathering-nkc49-p6m54\" (UID: \"09f5d7e2-51b8-4153-89c5-674d67065762\") " pod="openshift-insights/periodic-gathering-nkc49-p6m54" Oct 30 00:04:40.690550 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.690530 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/09f5d7e2-51b8-4153-89c5-674d67065762-archives-path\") pod \"periodic-gathering-nkc49-p6m54\" (UID: \"09f5d7e2-51b8-4153-89c5-674d67065762\") " pod="openshift-insights/periodic-gathering-nkc49-p6m54" Oct 30 00:04:40.690597 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.690563 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09f5d7e2-51b8-4153-89c5-674d67065762-trusted-ca-bundle\") pod \"periodic-gathering-nkc49-p6m54\" (UID: \"09f5d7e2-51b8-4153-89c5-674d67065762\") " pod="openshift-insights/periodic-gathering-nkc49-p6m54" Oct 30 00:04:40.714356 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.714331 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgkdb\" (UniqueName: \"kubernetes.io/projected/09f5d7e2-51b8-4153-89c5-674d67065762-kube-api-access-lgkdb\") pod \"periodic-gathering-nkc49-p6m54\" (UID: \"09f5d7e2-51b8-4153-89c5-674d67065762\") " pod="openshift-insights/periodic-gathering-nkc49-p6m54" Oct 30 00:04:40.770974 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.770942 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-nkc49-p6m54" Oct 30 00:04:40.864310 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.864282 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xt8f2"] Oct 30 00:04:40.879213 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.879185 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xt8f2" Oct 30 00:04:40.887741 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.887577 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xt8f2"] Oct 30 00:04:40.893836 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.893745 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8gb5_9fa688d8-29ca-4c1f-926a-f1a4099e6b44/console-operator/1.log" Oct 30 00:04:40.895037 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.894391 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8gb5_9fa688d8-29ca-4c1f-926a-f1a4099e6b44/console-operator/0.log" Oct 30 00:04:40.895037 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.894431 2580 generic.go:356] "Generic (PLEG): container finished" podID="9fa688d8-29ca-4c1f-926a-f1a4099e6b44" containerID="1b5598b379728a399ec94fb1bd83609608ffec6b569f58caae7897424bbb8d5f" exitCode=255 Oct 30 00:04:40.895037 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.894553 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" event={"ID":"9fa688d8-29ca-4c1f-926a-f1a4099e6b44","Type":"ContainerDied","Data":"1b5598b379728a399ec94fb1bd83609608ffec6b569f58caae7897424bbb8d5f"} Oct 30 00:04:40.895037 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.894604 2580 scope.go:122] "RemoveContainer" containerID="dadbee92034e275ef20827808b6f9ee7bdfb4aa33030a53a09e30fd4c91474f6" Oct 30 00:04:40.900288 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.899494 2580 scope.go:122] "RemoveContainer" containerID="1b5598b379728a399ec94fb1bd83609608ffec6b569f58caae7897424bbb8d5f" Oct 30 00:04:40.900288 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:40.899704 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-z8gb5_openshift-console-operator(9fa688d8-29ca-4c1f-926a-f1a4099e6b44)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" podUID="9fa688d8-29ca-4c1f-926a-f1a4099e6b44" Oct 30 00:04:40.926188 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.926159 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mpg4t" event={"ID":"de19ca9c-b22d-49a4-a5a6-a5f3a61bf892","Type":"ContainerStarted","Data":"e992665e8ea27c23980ae24bde4afa026558b1c46ef258420c6f9bf3eb98f51e"} Oct 30 00:04:40.935878 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.935847 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6d24p" event={"ID":"8a82abd0-5057-4361-9b0d-7a4f23ae97d1","Type":"ContainerStarted","Data":"4cbd23ae981f19cba837871a1c38bcf515259628b6992cfd909fbeb1b3742a4a"} Oct 30 00:04:40.982946 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.982875 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-l5flb\"" Oct 30 00:04:40.983122 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.983070 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 00:04:40.983181 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.983122 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 00:04:40.992487 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:40.992458 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzmtr\" (UniqueName: \"kubernetes.io/projected/8d95c49a-0861-4878-b10f-6b51c391b19e-kube-api-access-dzmtr\") pod \"migrator-66df5d496b-xt8f2\" (UID: \"8d95c49a-0861-4878-b10f-6b51c391b19e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xt8f2" Oct 30 00:04:41.009439 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:41.009162 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6d24p" podStartSLOduration=216.149901708 podStartE2EDuration="3m42.009144318s" totalImagesPullingTime="5.85924261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:00:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:41.007246785 +0000 UTC m=+37.949939626" watchObservedRunningTime="2026-10-30 00:04:41.009144318 +0000 UTC m=+37.951837162" Oct 30 00:04:41.094209 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:41.094172 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzmtr\" (UniqueName: \"kubernetes.io/projected/8d95c49a-0861-4878-b10f-6b51c391b19e-kube-api-access-dzmtr\") pod \"migrator-66df5d496b-xt8f2\" (UID: \"8d95c49a-0861-4878-b10f-6b51c391b19e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xt8f2" Oct 30 00:04:41.094383 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:41.094223 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12aceb3a-2476-4efc-94d8-e6ec78cfc680-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:41.094452 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:41.094415 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12aceb3a-2476-4efc-94d8-e6ec78cfc680-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:41.098570 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:41.097690 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:04:41.098570 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:41.097756 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12aceb3a-2476-4efc-94d8-e6ec78cfc680-insights-runtime-extractor-tls podName:12aceb3a-2476-4efc-94d8-e6ec78cfc680 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:42.097736748 +0000 UTC m=+39.040429582 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/12aceb3a-2476-4efc-94d8-e6ec78cfc680-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dd8wx" (UID: "12aceb3a-2476-4efc-94d8-e6ec78cfc680") : secret "insights-runtime-extractor-tls" not found Oct 30 00:04:41.098570 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:41.098098 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 00:04:41.098570 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:41.098144 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12aceb3a-2476-4efc-94d8-e6ec78cfc680-kube-rbac-proxy-cm podName:12aceb3a-2476-4efc-94d8-e6ec78cfc680 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:42.098129708 +0000 UTC m=+39.040822541 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/12aceb3a-2476-4efc-94d8-e6ec78cfc680-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dd8wx" (UID: "12aceb3a-2476-4efc-94d8-e6ec78cfc680") : configmap "kube-rbac-proxy" not found Oct 30 00:04:41.112367 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:41.112041 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mpg4t" podStartSLOduration=2.539497924 podStartE2EDuration="8.112021579s" totalImagesPullingTime="5.572523655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:04:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:41.049855887 +0000 UTC m=+37.992548729" watchObservedRunningTime="2026-10-30 00:04:41.112021579 +0000 UTC m=+38.054714422" Oct 30 00:04:41.112367 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:41.112294 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-nkc49-p6m54"] Oct 30 00:04:41.116696 ip-10-0-143-120 kubenswrapper[2580]: W1030 00:04:41.116668 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f5d7e2_51b8_4153_89c5_674d67065762.slice/crio-115ab9e9fecc7cf88524a0ecdc1649c7caec7db2b46b6428070d63a5fbf22474 WatchSource:0}: Error finding container 115ab9e9fecc7cf88524a0ecdc1649c7caec7db2b46b6428070d63a5fbf22474: Status 404 returned error can't find the container with id 115ab9e9fecc7cf88524a0ecdc1649c7caec7db2b46b6428070d63a5fbf22474 Oct 30 00:04:41.123085 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:41.123065 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzmtr\" (UniqueName: \"kubernetes.io/projected/8d95c49a-0861-4878-b10f-6b51c391b19e-kube-api-access-dzmtr\") pod \"migrator-66df5d496b-xt8f2\" (UID: \"8d95c49a-0861-4878-b10f-6b51c391b19e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xt8f2" Oct 30 00:04:41.226670 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:41.226646 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xt8f2" Oct 30 00:04:41.568727 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:41.566065 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xt8f2"] Oct 30 00:04:41.810020 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:41.809977 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acc82f37-fad2-424d-89d0-5e5876cb75de-metrics-tls\") pod \"dns-default-52kw4\" (UID: \"acc82f37-fad2-424d-89d0-5e5876cb75de\") " pod="openshift-dns/dns-default-52kw4" Oct 30 00:04:41.810199 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:41.810053 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7m2x2\" (UID: \"0d3be8a7-773b-42e1-a9a1-60d7df717301\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7m2x2" Oct 30 00:04:41.810199 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:41.810087 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l25ts\" (UID: \"18a4ebbe-c390-49d8-99e7-858bfaa3b834\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l25ts" Oct 30 00:04:41.810319 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:41.810210 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:04:41.810319 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:41.810263 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-tls\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:41.810319 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:41.810304 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aff3a44b-6cad-449e-8733-f85e9899ecc3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nqw98\" (UID: \"aff3a44b-6cad-449e-8733-f85e9899ecc3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nqw98" Oct 30 00:04:41.810466 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:41.810370 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d080b8e-cca6-4040-830e-cc9e9e25c50a-service-ca-bundle\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:41.810466 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:41.810406 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-metrics-certs\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:41.810466 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:41.810445 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-registry-tls\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:41.810612 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:41.810575 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:04:41.810612 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:41.810587 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dcddbb874-9zx5v: secret "image-registry-tls" not found Oct 30 00:04:41.810703 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:41.810635 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-registry-tls podName:526cc561-1bbd-4079-8a2f-326f939753e0 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:49.810618711 +0000 UTC m=+46.753311533 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-registry-tls") pod "image-registry-7dcddbb874-9zx5v" (UID: "526cc561-1bbd-4079-8a2f-326f939753e0") : secret "image-registry-tls" not found Oct 30 00:04:41.810703 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:41.810693 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 00:04:41.810806 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:41.810723 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-tls podName:18a4ebbe-c390-49d8-99e7-858bfaa3b834 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:49.810713089 +0000 UTC m=+46.753405922 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-l25ts" (UID: "18a4ebbe-c390-49d8-99e7-858bfaa3b834") : secret "samples-operator-tls" not found Oct 30 00:04:41.810806 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:41.810779 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:04:41.810806 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:41.810789 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c8c8757f-tspqd: secret "image-registry-tls" not found Oct 30 00:04:41.810982 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:41.810817 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-tls podName:fd8f9a4a-5b77-4a52-9ea9-a12245c00489 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:49.810808161 +0000 UTC m=+46.753500994 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-tls") pod "image-registry-7c8c8757f-tspqd" (UID: "fd8f9a4a-5b77-4a52-9ea9-a12245c00489") : secret "image-registry-tls" not found Oct 30 00:04:41.810982 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:41.810852 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d080b8e-cca6-4040-830e-cc9e9e25c50a-service-ca-bundle podName:9d080b8e-cca6-4040-830e-cc9e9e25c50a nodeName:}" failed. No retries permitted until 2026-10-30 00:04:49.810833394 +0000 UTC m=+46.753526228 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9d080b8e-cca6-4040-830e-cc9e9e25c50a-service-ca-bundle") pod "router-default-645c8fd54c-hp29d" (UID: "9d080b8e-cca6-4040-830e-cc9e9e25c50a") : configmap references non-existent config key: service-ca.crt Oct 30 00:04:41.810982 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:41.810873 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 00:04:41.810982 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:41.810900 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-metrics-certs podName:9d080b8e-cca6-4040-830e-cc9e9e25c50a nodeName:}" failed. No retries permitted until 2026-10-30 00:04:49.810890532 +0000 UTC m=+46.753583386 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-metrics-certs") pod "router-default-645c8fd54c-hp29d" (UID: "9d080b8e-cca6-4040-830e-cc9e9e25c50a") : secret "router-metrics-certs-default" not found Oct 30 00:04:41.810982 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:41.810965 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 00:04:41.811215 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:41.811010 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 00:04:41.811215 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:41.810967 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acc82f37-fad2-424d-89d0-5e5876cb75de-metrics-tls podName:acc82f37-fad2-424d-89d0-5e5876cb75de nodeName:}" failed. No retries permitted until 2026-10-30 00:04:49.810952282 +0000 UTC m=+46.753645144 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acc82f37-fad2-424d-89d0-5e5876cb75de-metrics-tls") pod "dns-default-52kw4" (UID: "acc82f37-fad2-424d-89d0-5e5876cb75de") : secret "dns-default-metrics-tls" not found Oct 30 00:04:41.811215 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:41.811060 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff3a44b-6cad-449e-8733-f85e9899ecc3-cluster-monitoring-operator-tls podName:aff3a44b-6cad-449e-8733-f85e9899ecc3 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:49.811050989 +0000 UTC m=+46.753743814 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/aff3a44b-6cad-449e-8733-f85e9899ecc3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nqw98" (UID: "aff3a44b-6cad-449e-8733-f85e9899ecc3") : secret "cluster-monitoring-operator-tls" not found Oct 30 00:04:41.811215 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:41.811074 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert podName:0d3be8a7-773b-42e1-a9a1-60d7df717301 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:49.811064913 +0000 UTC m=+46.753757740 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7m2x2" (UID: "0d3be8a7-773b-42e1-a9a1-60d7df717301") : secret "networking-console-plugin-cert" not found Oct 30 00:04:41.941033 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:41.940949 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-glf29" event={"ID":"32309642-7cba-4361-83b3-4680659c0544","Type":"ContainerStarted","Data":"6e35dbc41676c366489c06541e1d588ceea26514eb0a04728f32b6954287968e"} Oct 30 00:04:41.943086 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:41.943050 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-nkc49-p6m54" event={"ID":"09f5d7e2-51b8-4153-89c5-674d67065762","Type":"ContainerStarted","Data":"56f23de064fa0cb6e1bd391a754c29fac230383af672fc16c364bcc6cab65f7d"} Oct 30 00:04:41.943217 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:41.943098 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-nkc49-p6m54" event={"ID":"09f5d7e2-51b8-4153-89c5-674d67065762","Type":"ContainerStarted","Data":"115ab9e9fecc7cf88524a0ecdc1649c7caec7db2b46b6428070d63a5fbf22474"} Oct 30 00:04:41.944879 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:41.944860 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8gb5_9fa688d8-29ca-4c1f-926a-f1a4099e6b44/console-operator/1.log" Oct 30 00:04:41.945206 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:41.945183 2580 scope.go:122] "RemoveContainer" containerID="1b5598b379728a399ec94fb1bd83609608ffec6b569f58caae7897424bbb8d5f" Oct 30 00:04:41.945394 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:41.945369 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-z8gb5_openshift-console-operator(9fa688d8-29ca-4c1f-926a-f1a4099e6b44)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" podUID="9fa688d8-29ca-4c1f-926a-f1a4099e6b44" Oct 30 00:04:41.946202 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:41.946183 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xt8f2" event={"ID":"8d95c49a-0861-4878-b10f-6b51c391b19e","Type":"ContainerStarted","Data":"dbc06f0c0d480c5387e4cf6ad3b840b61fcc9f9eec5cd5ca71fb4f9d0e28835a"} Oct 30 00:04:41.985899 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:41.985841 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-glf29" podStartSLOduration=210.685408096 podStartE2EDuration="3m36.985822606s" totalImagesPullingTime="6.30041451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:01:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:41.984541855 +0000 UTC m=+38.927234713" watchObservedRunningTime="2026-10-30 00:04:41.985822606 +0000 UTC m=+38.928515453" Oct 30 00:04:42.029131 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:42.029076 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-nkc49-p6m54" podStartSLOduration=2.02906167 podStartE2EDuration="2.02906167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:42.028048553 +0000 UTC m=+38.970741396" watchObservedRunningTime="2026-10-30 00:04:42.02906167 +0000 UTC m=+38.971754512" Oct 30 00:04:42.113253 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:42.113214 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12aceb3a-2476-4efc-94d8-e6ec78cfc680-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:42.113408 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:42.113380 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12aceb3a-2476-4efc-94d8-e6ec78cfc680-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:42.113481 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:42.113423 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:04:42.113539 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:42.113486 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12aceb3a-2476-4efc-94d8-e6ec78cfc680-insights-runtime-extractor-tls podName:12aceb3a-2476-4efc-94d8-e6ec78cfc680 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:44.113467366 +0000 UTC m=+41.056160185 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/12aceb3a-2476-4efc-94d8-e6ec78cfc680-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dd8wx" (UID: "12aceb3a-2476-4efc-94d8-e6ec78cfc680") : secret "insights-runtime-extractor-tls" not found Oct 30 00:04:42.113891 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:42.113871 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 00:04:42.114002 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:42.113947 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12aceb3a-2476-4efc-94d8-e6ec78cfc680-kube-rbac-proxy-cm podName:12aceb3a-2476-4efc-94d8-e6ec78cfc680 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:44.113928124 +0000 UTC m=+41.056620959 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/12aceb3a-2476-4efc-94d8-e6ec78cfc680-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dd8wx" (UID: "12aceb3a-2476-4efc-94d8-e6ec78cfc680") : configmap "kube-rbac-proxy" not found Oct 30 00:04:42.951468 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:42.951428 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sxvqx" event={"ID":"a20efc24-e4cb-4075-911e-bfc43dcfc20b","Type":"ContainerStarted","Data":"f0e4e39641593a60043922e187e5496d9dd891b29a127fd34ea0cc89d0c5da5a"} Oct 30 00:04:42.951968 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:42.951636 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sxvqx" Oct 30 00:04:42.998237 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:42.998177 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-sxvqx" podStartSLOduration=34.974549937 podStartE2EDuration="38.99815757s" totalImagesPullingTime="4.023607633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:04:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:42.997187906 +0000 UTC m=+39.939880748" watchObservedRunningTime="2026-10-30 00:04:42.99815757 +0000 UTC m=+39.940850412" Oct 30 00:04:43.957379 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:43.957344 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xt8f2" event={"ID":"8d95c49a-0861-4878-b10f-6b51c391b19e","Type":"ContainerStarted","Data":"4e24a90408af7f06528d46d0e81492f71f6bea7df1663f36977e2612e06d7c0c"} Oct 30 00:04:43.999343 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:43.999283 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xt8f2" podStartSLOduration=2.372844039 podStartE2EDuration="3.999264883s" totalImagesPullingTime="1.626420844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:04:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:04:43.997892731 +0000 UTC m=+40.940585575" watchObservedRunningTime="2026-10-30 00:04:43.999264883 +0000 UTC m=+40.941957727" Oct 30 00:04:44.136559 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:44.136520 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12aceb3a-2476-4efc-94d8-e6ec78cfc680-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:44.136731 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:44.136598 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12aceb3a-2476-4efc-94d8-e6ec78cfc680-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:44.136731 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:44.136692 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 00:04:44.136845 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:44.136730 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:04:44.136845 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:44.136760 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12aceb3a-2476-4efc-94d8-e6ec78cfc680-kube-rbac-proxy-cm podName:12aceb3a-2476-4efc-94d8-e6ec78cfc680 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:48.136742501 +0000 UTC m=+45.079435328 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/12aceb3a-2476-4efc-94d8-e6ec78cfc680-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dd8wx" (UID: "12aceb3a-2476-4efc-94d8-e6ec78cfc680") : configmap "kube-rbac-proxy" not found Oct 30 00:04:44.136845 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:44.136784 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12aceb3a-2476-4efc-94d8-e6ec78cfc680-insights-runtime-extractor-tls podName:12aceb3a-2476-4efc-94d8-e6ec78cfc680 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:48.136771814 +0000 UTC m=+45.079464638 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/12aceb3a-2476-4efc-94d8-e6ec78cfc680-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dd8wx" (UID: "12aceb3a-2476-4efc-94d8-e6ec78cfc680") : secret "insights-runtime-extractor-tls" not found Oct 30 00:04:44.273096 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:44.273060 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" Oct 30 00:04:44.273258 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:44.273121 2580 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" Oct 30 00:04:44.273511 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:44.273493 2580 scope.go:122] "RemoveContainer" containerID="1b5598b379728a399ec94fb1bd83609608ffec6b569f58caae7897424bbb8d5f" Oct 30 00:04:44.273712 ip-10-0-143-120 kubenswrapper[2580]: E1030 00:04:44.273690 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-z8gb5_openshift-console-operator(9fa688d8-29ca-4c1f-926a-f1a4099e6b44)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" podUID="9fa688d8-29ca-4c1f-926a-f1a4099e6b44" Oct 30 00:04:44.479605 ip-10-0-143-120 kubenswrapper[2580]: I1030 00:04:44.479579 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b54v5_a9e7a0c8-43b3-4f8c-8870-505edace7f78/dns-node-resolver/0.log" Oct 30 00:04:44.738713 ip-10-0-143-120 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 00:04:44.750284 ip-10-0-143-120 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 00:04:44.750444 ip-10-0-143-120 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 00:04:44.750498 ip-10-0-143-120 systemd[1]: kubelet.service: Consumed 3.046s CPU time. Oct 30 00:04:44.764370 ip-10-0-143-120 systemd[1]: Starting Kubernetes Kubelet... Oct 30 00:04:44.836126 ip-10-0-143-120 kubenswrapper[5001]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:04:44.836126 ip-10-0-143-120 kubenswrapper[5001]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 00:04:44.836126 ip-10-0-143-120 kubenswrapper[5001]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:04:44.836126 ip-10-0-143-120 kubenswrapper[5001]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:04:44.838091 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838008 5001 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:04:44.838091 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838090 5001 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:04:44.838161 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838095 5001 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:04:44.838161 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838098 5001 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:04:44.838161 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838101 5001 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:04:44.838161 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838104 5001 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:04:44.838161 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838107 5001 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:04:44.838161 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838110 5001 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:04:44.838161 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838112 5001 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:04:44.838161 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838115 5001 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:04:44.838161 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838118 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:04:44.838161 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838120 5001 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:04:44.838161 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838123 5001 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:04:44.838161 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838126 5001 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:04:44.838161 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838129 5001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:04:44.838161 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838131 5001 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:04:44.838161 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838134 5001 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:04:44.838161 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838137 5001 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:04:44.838161 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838140 5001 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:04:44.838161 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838142 5001 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:04:44.838161 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838145 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:04:44.838161 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838148 5001 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:04:44.838663 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838150 5001 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:04:44.838663 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838153 5001 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:04:44.838663 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838155 5001 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:04:44.838663 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838158 5001 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:04:44.838663 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838160 5001 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:04:44.838663 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838163 5001 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:04:44.838663 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838173 5001 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:04:44.838663 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838177 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:04:44.838663 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838179 5001 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:04:44.838663 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838182 5001 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:04:44.838663 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838184 5001 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:04:44.838663 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838187 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:04:44.838663 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838189 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:04:44.838663 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838192 5001 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:04:44.838663 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838194 5001 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:04:44.838663 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838197 5001 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:04:44.838663 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838199 5001 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:04:44.838663 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838202 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:04:44.838663 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838205 5001 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:04:44.838663 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838207 5001 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:04:44.839189 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838210 5001 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:04:44.839189 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838212 5001 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:04:44.839189 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838215 5001 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:04:44.839189 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838217 5001 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:04:44.839189 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838220 5001 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:04:44.839189 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838223 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:04:44.839189 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838226 5001 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:04:44.839189 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838229 5001 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:04:44.839189 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838232 5001 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:04:44.839189 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838234 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:04:44.839189 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838238 5001 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:04:44.839189 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838240 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:04:44.839189 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838243 5001 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:04:44.839189 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838246 5001 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:04:44.839189 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838248 5001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:04:44.839189 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838251 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:04:44.839189 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838254 5001 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:04:44.839189 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838256 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:04:44.839189 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838259 5001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:04:44.839694 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838261 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:04:44.839694 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838264 5001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:04:44.839694 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838266 5001 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:04:44.839694 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838271 5001 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:04:44.839694 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838275 5001 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:04:44.839694 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838279 5001 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:04:44.839694 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838281 5001 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:04:44.839694 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838285 5001 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:04:44.839694 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838288 5001 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:04:44.839694 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838290 5001 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:04:44.839694 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838293 5001 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:04:44.839694 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838295 5001 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:04:44.839694 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838299 5001 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:04:44.839694 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838303 5001 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:04:44.839694 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838306 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:04:44.839694 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838309 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:04:44.839694 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838311 5001 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:04:44.839694 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838314 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:04:44.839694 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838317 5001 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:04:44.840192 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838320 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:04:44.840192 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838322 5001 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:04:44.840192 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838325 5001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:04:44.840192 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838328 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:04:44.840192 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838331 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:04:44.840192 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838334 5001 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:04:44.840192 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838337 5001 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:04:44.840192 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838339 5001 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:04:44.840192 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838342 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:04:44.840192 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838346 5001 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:04:44.840192 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838349 5001 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:04:44.840192 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838351 5001 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:04:44.840192 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838354 5001 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:04:44.840192 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838356 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:04:44.840192 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838359 5001 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:04:44.840192 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838362 5001 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:04:44.840192 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838364 5001 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:04:44.840192 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838367 5001 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:04:44.840192 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838369 5001 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:04:44.840671 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838372 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:04:44.840671 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838374 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:04:44.840671 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838377 5001 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:04:44.840671 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838379 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:04:44.840671 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838382 5001 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:04:44.840671 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838385 5001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:04:44.840671 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.838387 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:04:44.840671 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839139 5001 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:04:44.840671 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839146 5001 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:04:44.840671 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839150 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:04:44.840671 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839152 5001 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:04:44.840671 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839155 5001 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:04:44.840671 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839158 5001 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:04:44.840671 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839161 5001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:04:44.840671 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839164 5001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:04:44.840671 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839172 5001 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:04:44.840671 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839175 5001 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:04:44.840671 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839177 5001 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:04:44.840671 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839180 5001 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:04:44.841154 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839183 5001 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:04:44.841154 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839185 5001 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:04:44.841154 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839188 5001 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:04:44.841154 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839191 5001 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:04:44.841154 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839194 5001 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:04:44.841154 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839197 5001 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:04:44.841154 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839200 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:04:44.841154 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839202 5001 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:04:44.841154 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839207 5001 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:04:44.841154 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839211 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:04:44.841154 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839214 5001 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:04:44.841154 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839217 5001 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:04:44.841154 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839220 5001 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:04:44.841154 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839223 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:04:44.841154 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839226 5001 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:04:44.841154 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839228 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:04:44.841154 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839232 5001 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:04:44.841154 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839236 5001 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:04:44.841612 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839239 5001 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:04:44.841612 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839242 5001 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:04:44.841612 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839246 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:04:44.841612 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839249 5001 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:04:44.841612 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839251 5001 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:04:44.841612 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839254 5001 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:04:44.841612 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839257 5001 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:04:44.841612 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839259 5001 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:04:44.841612 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839262 5001 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:04:44.841612 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839264 5001 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:04:44.841612 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839269 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:04:44.841612 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839271 5001 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:04:44.841612 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839274 5001 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:04:44.841612 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839277 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:04:44.841612 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839279 5001 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:04:44.841612 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839282 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:04:44.841612 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839284 5001 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:04:44.841612 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839288 5001 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:04:44.841612 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839290 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:04:44.841612 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839293 5001 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:04:44.842125 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839296 5001 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:04:44.842125 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839298 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:04:44.842125 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839301 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:04:44.842125 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839304 5001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:04:44.842125 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839306 5001 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:04:44.842125 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839309 5001 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:04:44.842125 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839311 5001 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:04:44.842125 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839314 5001 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:04:44.842125 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839317 5001 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:04:44.842125 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839319 5001 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:04:44.842125 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839322 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:04:44.842125 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839325 5001 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:04:44.842125 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839327 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:04:44.842125 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839330 5001 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:04:44.842125 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839332 5001 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:04:44.842125 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839335 5001 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:04:44.842125 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839337 5001 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:04:44.842125 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839340 5001 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:04:44.842125 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839342 5001 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:04:44.842125 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839345 5001 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:04:44.842643 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839347 5001 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:04:44.842643 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839350 5001 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:04:44.842643 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839353 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:04:44.842643 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839357 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:04:44.842643 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839359 5001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:04:44.842643 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839362 5001 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:04:44.842643 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839365 5001 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:04:44.842643 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839368 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:04:44.842643 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839370 5001 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:04:44.842643 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839373 5001 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:04:44.842643 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839375 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:04:44.842643 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839378 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:04:44.842643 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839380 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:04:44.842643 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839383 5001 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:04:44.842643 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839386 5001 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:04:44.842643 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839388 5001 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:04:44.842643 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839391 5001 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:04:44.842643 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839394 5001 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:04:44.842643 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839397 5001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:04:44.842643 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839400 5001 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:04:44.843143 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839402 5001 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:04:44.843143 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839405 5001 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:04:44.843143 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839407 5001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:04:44.843143 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839410 5001 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:04:44.843143 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839412 5001 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:04:44.843143 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839415 5001 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:04:44.843143 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839418 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:04:44.843143 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839420 5001 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:04:44.843143 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839423 5001 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:04:44.843143 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839425 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:04:44.843143 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839428 5001 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:04:44.843143 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839431 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:04:44.843143 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839433 5001 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:04:44.843143 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839436 5001 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:04:44.843143 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839439 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:04:44.843143 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839443 5001 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:04:44.843143 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839932 5001 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:04:44.843143 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839937 5001 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:04:44.843143 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839940 5001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:04:44.843143 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839944 5001 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:04:44.843624 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839947 5001 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:04:44.843624 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839950 5001 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:04:44.843624 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839952 5001 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:04:44.843624 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839955 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:04:44.843624 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839958 5001 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:04:44.843624 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839961 5001 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:04:44.843624 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839964 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:04:44.843624 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839966 5001 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:04:44.843624 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839969 5001 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:04:44.843624 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839972 5001 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:04:44.843624 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839974 5001 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:04:44.843624 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839977 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:04:44.843624 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839980 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:04:44.843624 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839982 5001 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:04:44.843624 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839985 5001 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:04:44.843624 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839987 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:04:44.843624 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839990 5001 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:04:44.843624 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839992 5001 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:04:44.843624 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839995 5001 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:04:44.844160 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.839997 5001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:04:44.844160 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840001 5001 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:04:44.844160 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840005 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:04:44.844160 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840009 5001 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:04:44.844160 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840011 5001 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:04:44.844160 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840015 5001 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:04:44.844160 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840018 5001 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:04:44.844160 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840022 5001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:04:44.844160 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840025 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:04:44.844160 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840028 5001 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:04:44.844160 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840031 5001 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:04:44.844160 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840033 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:04:44.844160 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840036 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:04:44.844160 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840039 5001 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:04:44.844160 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840041 5001 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:04:44.844160 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840044 5001 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:04:44.844160 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840046 5001 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:04:44.844160 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840049 5001 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:04:44.844160 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840051 5001 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:04:44.844635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840054 5001 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:04:44.844635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840057 5001 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:04:44.844635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840059 5001 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:04:44.844635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840062 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:04:44.844635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840065 5001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:04:44.844635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840067 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:04:44.844635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840070 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:04:44.844635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840073 5001 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:04:44.844635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840076 5001 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:04:44.844635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840079 5001 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:04:44.844635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840081 5001 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:04:44.844635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840084 5001 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:04:44.844635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840087 5001 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:04:44.844635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840089 5001 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:04:44.844635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840091 5001 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:04:44.844635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840094 5001 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:04:44.844635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840097 5001 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:04:44.844635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840099 5001 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:04:44.844635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840104 5001 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:04:44.844635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840107 5001 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:04:44.845151 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840110 5001 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:04:44.845151 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840113 5001 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:04:44.845151 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840116 5001 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:04:44.845151 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840121 5001 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:04:44.845151 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840124 5001 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:04:44.845151 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840127 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:04:44.845151 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840129 5001 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:04:44.845151 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840132 5001 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:04:44.845151 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840135 5001 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:04:44.845151 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840137 5001 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:04:44.845151 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840140 5001 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:04:44.845151 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840142 5001 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:04:44.845151 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840145 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:04:44.845151 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840148 5001 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:04:44.845151 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840150 5001 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:04:44.845151 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840153 5001 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:04:44.845151 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840156 5001 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:04:44.845151 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840158 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:04:44.845151 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840161 5001 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:04:44.845645 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840164 5001 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:04:44.845645 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840166 5001 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:04:44.845645 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840169 5001 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:04:44.845645 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840171 5001 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:04:44.845645 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840174 5001 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:04:44.845645 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840177 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:04:44.845645 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840179 5001 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:04:44.845645 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840182 5001 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:04:44.845645 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840184 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:04:44.845645 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840187 5001 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:04:44.845645 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840191 5001 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:04:44.845645 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840196 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:04:44.845645 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840199 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:04:44.845645 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840203 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:04:44.845645 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840206 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:04:44.845645 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840209 5001 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:04:44.845645 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840211 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:04:44.845645 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840215 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:04:44.845645 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840218 5001 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:04:44.846113 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840220 5001 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:04:44.846113 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840223 5001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:04:44.846113 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840225 5001 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:04:44.846113 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840228 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:04:44.846113 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840231 5001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:04:44.846113 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840233 5001 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:04:44.846113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840330 5001 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 00:04:44.846113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840337 5001 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 00:04:44.846113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840343 5001 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 00:04:44.846113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840348 5001 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 00:04:44.846113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840352 5001 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 00:04:44.846113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840355 5001 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 00:04:44.846113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840359 5001 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 00:04:44.846113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840363 5001 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 00:04:44.846113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840366 5001 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 00:04:44.846113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840369 5001 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 00:04:44.846113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840373 5001 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 00:04:44.846113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840376 5001 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 00:04:44.846113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840379 5001 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 00:04:44.846113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840382 5001 flags.go:64] FLAG: --cgroup-root="" Oct 30 00:04:44.846113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840385 5001 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 00:04:44.846113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840388 5001 flags.go:64] FLAG: --client-ca-file="" Oct 30 00:04:44.846699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840391 5001 flags.go:64] FLAG: --cloud-provider="external" Oct 30 00:04:44.846699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840394 5001 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 00:04:44.846699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840401 5001 flags.go:64] FLAG: --cluster-domain="" Oct 30 00:04:44.846699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840405 5001 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 00:04:44.846699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840408 5001 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 00:04:44.846699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840411 5001 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 00:04:44.846699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840414 5001 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 00:04:44.846699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840418 5001 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 00:04:44.846699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840421 5001 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 00:04:44.846699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840425 5001 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 00:04:44.846699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840428 5001 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 00:04:44.846699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840431 5001 flags.go:64] FLAG: --contention-profiling="false" Oct 30 00:04:44.846699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840434 5001 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 00:04:44.846699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840438 5001 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 00:04:44.846699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840441 5001 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 00:04:44.846699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840443 5001 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 00:04:44.846699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840448 5001 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 00:04:44.846699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840451 5001 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 00:04:44.846699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840454 5001 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 00:04:44.846699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840457 5001 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 00:04:44.846699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840460 5001 flags.go:64] FLAG: --enable-server="true" Oct 30 00:04:44.846699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840463 5001 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 00:04:44.846699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840467 5001 flags.go:64] FLAG: --event-burst="100" Oct 30 00:04:44.846699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840470 5001 flags.go:64] FLAG: --event-qps="50" Oct 30 00:04:44.846699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840473 5001 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 00:04:44.847304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840476 5001 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 00:04:44.847304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840479 5001 flags.go:64] FLAG: --eviction-hard="" Oct 30 00:04:44.847304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840484 5001 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 00:04:44.847304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840487 5001 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 00:04:44.847304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840490 5001 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 00:04:44.847304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840493 5001 flags.go:64] FLAG: --eviction-soft="" Oct 30 00:04:44.847304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840496 5001 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 00:04:44.847304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840499 5001 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 00:04:44.847304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840503 5001 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 00:04:44.847304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840507 5001 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 00:04:44.847304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840510 5001 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 00:04:44.847304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840513 5001 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 00:04:44.847304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840516 5001 flags.go:64] FLAG: --feature-gates="" Oct 30 00:04:44.847304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840520 5001 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 00:04:44.847304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840523 5001 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 00:04:44.847304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840526 5001 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 00:04:44.847304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840530 5001 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 00:04:44.847304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840533 5001 flags.go:64] FLAG: --healthz-port="10248" Oct 30 00:04:44.847304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840537 5001 flags.go:64] FLAG: --help="false" Oct 30 00:04:44.847304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840540 5001 flags.go:64] FLAG: --hostname-override="ip-10-0-143-120.ec2.internal" Oct 30 00:04:44.847304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840543 5001 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 00:04:44.847304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840546 5001 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 00:04:44.847304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840549 5001 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 00:04:44.847857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840553 5001 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 00:04:44.847857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840556 5001 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 00:04:44.847857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840559 5001 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 00:04:44.847857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840562 5001 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 00:04:44.847857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840565 5001 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 00:04:44.847857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840568 5001 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 00:04:44.847857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840571 5001 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 00:04:44.847857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840574 5001 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 00:04:44.847857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840577 5001 flags.go:64] FLAG: --kube-reserved="" Oct 30 00:04:44.847857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840580 5001 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 00:04:44.847857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840583 5001 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 00:04:44.847857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840586 5001 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 00:04:44.847857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840589 5001 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 00:04:44.847857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840592 5001 flags.go:64] FLAG: --lock-file="" Oct 30 00:04:44.847857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840595 5001 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 00:04:44.847857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840598 5001 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 00:04:44.847857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840601 5001 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 00:04:44.847857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840606 5001 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 00:04:44.847857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840611 5001 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 00:04:44.847857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840614 5001 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 00:04:44.847857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840617 5001 flags.go:64] FLAG: --logging-format="text" Oct 30 00:04:44.847857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840620 5001 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 00:04:44.847857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840623 5001 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 00:04:44.847857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840626 5001 flags.go:64] FLAG: --manifest-url="" Oct 30 00:04:44.848477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840629 5001 flags.go:64] FLAG: --manifest-url-header="" Oct 30 00:04:44.848477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840633 5001 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 00:04:44.848477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840636 5001 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 00:04:44.848477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840646 5001 flags.go:64] FLAG: --max-pods="110" Oct 30 00:04:44.848477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840649 5001 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 00:04:44.848477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840652 5001 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 00:04:44.848477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840655 5001 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 00:04:44.848477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840658 5001 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 00:04:44.848477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840661 5001 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 00:04:44.848477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840664 5001 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 00:04:44.848477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840667 5001 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 00:04:44.848477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840674 5001 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 00:04:44.848477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840677 5001 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 00:04:44.848477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840680 5001 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 00:04:44.848477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840685 5001 flags.go:64] FLAG: --pod-cidr="" Oct 30 00:04:44.848477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840688 5001 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 00:04:44.848477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840691 5001 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 00:04:44.848477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840694 5001 flags.go:64] FLAG: --pods-per-core="0" Oct 30 00:04:44.848477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840697 5001 flags.go:64] FLAG: --port="10250" Oct 30 00:04:44.848477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840700 5001 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 00:04:44.848477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840703 5001 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d2a8f9d9ac765af2" Oct 30 00:04:44.848477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840706 5001 flags.go:64] FLAG: --qos-reserved="" Oct 30 00:04:44.848477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840709 5001 flags.go:64] FLAG: --read-only-port="10255" Oct 30 00:04:44.848477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840712 5001 flags.go:64] FLAG: --register-node="true" Oct 30 00:04:44.848477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840715 5001 flags.go:64] FLAG: --register-with-taints="" Oct 30 00:04:44.848477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840719 5001 flags.go:64] FLAG: --registry-burst="10" Oct 30 00:04:44.849508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840723 5001 flags.go:64] FLAG: --registry-qps="5" Oct 30 00:04:44.849508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840726 5001 flags.go:64] FLAG: --reserved-cpus="" Oct 30 00:04:44.849508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840729 5001 flags.go:64] FLAG: --reserved-memory="" Oct 30 00:04:44.849508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840733 5001 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 00:04:44.849508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840736 5001 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 00:04:44.849508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840739 5001 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 00:04:44.849508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840741 5001 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 00:04:44.849508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840745 5001 flags.go:64] FLAG: --runonce="false" Oct 30 00:04:44.849508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840748 5001 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 00:04:44.849508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840751 5001 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 00:04:44.849508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840759 5001 flags.go:64] FLAG: --seccomp-default="false" Oct 30 00:04:44.849508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840763 5001 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 00:04:44.849508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840766 5001 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 00:04:44.849508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840769 5001 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 00:04:44.849508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840772 5001 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 00:04:44.849508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840775 5001 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 00:04:44.849508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840778 5001 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 00:04:44.849508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840781 5001 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 00:04:44.849508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840784 5001 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 00:04:44.849508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840787 5001 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 00:04:44.849508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840791 5001 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 00:04:44.849508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840794 5001 flags.go:64] FLAG: --system-cgroups="" Oct 30 00:04:44.849508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840797 5001 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 00:04:44.849508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840802 5001 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 00:04:44.849508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840805 5001 flags.go:64] FLAG: --tls-cert-file="" Oct 30 00:04:44.850346 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840808 5001 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 00:04:44.850346 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840815 5001 flags.go:64] FLAG: --tls-min-version="" Oct 30 00:04:44.850346 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840818 5001 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 00:04:44.850346 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840821 5001 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 00:04:44.850346 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840824 5001 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 00:04:44.850346 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840827 5001 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 00:04:44.850346 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840830 5001 flags.go:64] FLAG: --v="2" Oct 30 00:04:44.850346 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840835 5001 flags.go:64] FLAG: --version="false" Oct 30 00:04:44.850346 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840839 5001 flags.go:64] FLAG: --vmodule="" Oct 30 00:04:44.850346 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840843 5001 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 00:04:44.850346 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.840847 5001 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 00:04:44.850346 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.840999 5001 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:04:44.850346 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841003 5001 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:04:44.850346 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841007 5001 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:04:44.850346 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841010 5001 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:04:44.850346 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841013 5001 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:04:44.850346 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841016 5001 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:04:44.850346 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841019 5001 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:04:44.850346 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841027 5001 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:04:44.850346 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841030 5001 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:04:44.850346 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841033 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:04:44.850346 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841035 5001 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:04:44.850888 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841038 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:04:44.850888 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841041 5001 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:04:44.850888 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841043 5001 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:04:44.850888 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841046 5001 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:04:44.850888 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841049 5001 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:04:44.850888 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841053 5001 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:04:44.850888 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841056 5001 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:04:44.850888 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841059 5001 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:04:44.850888 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841062 5001 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:04:44.850888 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841064 5001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:04:44.850888 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841067 5001 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:04:44.850888 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841069 5001 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:04:44.850888 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841072 5001 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:04:44.850888 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841075 5001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:04:44.850888 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841077 5001 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:04:44.850888 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841080 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:04:44.850888 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841083 5001 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:04:44.850888 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841087 5001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:04:44.850888 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841089 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:04:44.850888 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841092 5001 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:04:44.850888 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841095 5001 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:04:44.851414 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841097 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:04:44.851414 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841100 5001 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:04:44.851414 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841102 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:04:44.851414 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841105 5001 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:04:44.851414 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841108 5001 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:04:44.851414 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841112 5001 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:04:44.851414 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841115 5001 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:04:44.851414 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841118 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:04:44.851414 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841128 5001 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:04:44.851414 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841131 5001 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:04:44.851414 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841133 5001 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:04:44.851414 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841136 5001 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:04:44.851414 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841139 5001 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:04:44.851414 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841141 5001 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:04:44.851414 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841144 5001 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:04:44.851414 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841146 5001 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:04:44.851414 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841151 5001 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:04:44.851414 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841154 5001 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:04:44.851414 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841156 5001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:04:44.851948 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841159 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:04:44.851948 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841162 5001 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:04:44.851948 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841165 5001 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:04:44.851948 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841168 5001 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:04:44.851948 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841170 5001 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:04:44.851948 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841173 5001 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:04:44.851948 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841176 5001 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:04:44.851948 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841178 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:04:44.851948 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841181 5001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:04:44.851948 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841185 5001 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:04:44.851948 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841188 5001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:04:44.851948 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841191 5001 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:04:44.851948 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841194 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:04:44.851948 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841196 5001 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:04:44.851948 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841199 5001 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:04:44.851948 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841202 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:04:44.851948 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841205 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:04:44.851948 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841208 5001 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:04:44.851948 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841211 5001 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:04:44.852635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841215 5001 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:04:44.852635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841219 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:04:44.852635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841222 5001 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:04:44.852635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841231 5001 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:04:44.852635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841234 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:04:44.852635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841236 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:04:44.852635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841240 5001 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:04:44.852635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841243 5001 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:04:44.852635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841246 5001 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:04:44.852635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841249 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:04:44.852635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841253 5001 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:04:44.852635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841255 5001 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:04:44.852635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841258 5001 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:04:44.852635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841261 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:04:44.852635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841264 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:04:44.852635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841266 5001 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:04:44.852635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841269 5001 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:04:44.852635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841271 5001 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:04:44.852635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841274 5001 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:04:44.853204 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841277 5001 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:04:44.853204 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841280 5001 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:04:44.853204 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841282 5001 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:04:44.853204 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841286 5001 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:04:44.853204 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841289 5001 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:04:44.853204 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841291 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:04:44.853204 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841294 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:04:44.853204 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841296 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:04:44.853204 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841299 5001 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:04:44.853204 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841302 5001 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:04:44.853204 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841304 5001 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:04:44.853204 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841307 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:04:44.853204 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841310 5001 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:04:44.853204 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841313 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:04:44.853204 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841315 5001 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:04:44.853204 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841318 5001 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:04:44.853204 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841320 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:04:44.853621 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.841347 5001 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:04:44.853621 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841388 5001 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:04:44.853621 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841393 5001 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:04:44.853621 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841396 5001 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:04:44.853621 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841399 5001 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:04:44.853621 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841401 5001 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:04:44.853621 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841404 5001 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:04:44.853621 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841408 5001 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:04:44.853621 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841411 5001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:04:44.853621 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841413 5001 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:04:44.853621 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841416 5001 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:04:44.853621 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841419 5001 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:04:44.853621 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841421 5001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:04:44.853621 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841424 5001 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:04:44.853621 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841427 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:04:44.853621 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841430 5001 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:04:44.853621 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841432 5001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:04:44.854059 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841435 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:04:44.854059 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841439 5001 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:04:44.854059 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841442 5001 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:04:44.854059 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841444 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:04:44.854059 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841447 5001 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:04:44.854059 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841450 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:04:44.854059 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841453 5001 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:04:44.854059 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841455 5001 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:04:44.854059 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841459 5001 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:04:44.854059 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841463 5001 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:04:44.854059 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841466 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:04:44.854059 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841469 5001 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:04:44.854059 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841472 5001 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:04:44.854059 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841475 5001 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:04:44.854059 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841477 5001 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:04:44.854059 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841480 5001 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:04:44.854059 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841483 5001 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:04:44.854059 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841485 5001 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:04:44.854059 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841488 5001 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:04:44.854652 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841491 5001 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:04:44.854652 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841493 5001 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:04:44.854652 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841496 5001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:04:44.854652 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841500 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:04:44.854652 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841503 5001 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:04:44.854652 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841506 5001 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:04:44.854652 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841508 5001 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:04:44.854652 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841511 5001 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:04:44.854652 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841514 5001 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:04:44.854652 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841516 5001 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:04:44.854652 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841519 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:04:44.854652 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841521 5001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:04:44.854652 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841524 5001 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:04:44.854652 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841527 5001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:04:44.854652 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841530 5001 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:04:44.854652 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841533 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:04:44.854652 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841536 5001 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:04:44.854652 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841539 5001 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:04:44.854652 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841542 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:04:44.855197 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841544 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:04:44.855197 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841547 5001 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:04:44.855197 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841550 5001 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:04:44.855197 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841554 5001 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:04:44.855197 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841557 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:04:44.855197 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841560 5001 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:04:44.855197 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841563 5001 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:04:44.855197 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841565 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:04:44.855197 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841568 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:04:44.855197 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841572 5001 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:04:44.855197 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841575 5001 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:04:44.855197 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841578 5001 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:04:44.855197 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841581 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:04:44.855197 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841583 5001 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:04:44.855197 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841586 5001 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:04:44.855197 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841588 5001 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:04:44.855197 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841592 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:04:44.855197 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841595 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:04:44.855197 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841598 5001 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:04:44.855676 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841601 5001 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:04:44.855676 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841604 5001 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:04:44.855676 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841607 5001 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:04:44.855676 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841609 5001 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:04:44.855676 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841612 5001 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:04:44.855676 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841614 5001 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:04:44.855676 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841617 5001 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:04:44.855676 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841620 5001 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:04:44.855676 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841625 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:04:44.855676 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841628 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:04:44.855676 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841630 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:04:44.855676 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841633 5001 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:04:44.855676 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841636 5001 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:04:44.855676 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841638 5001 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:04:44.855676 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841641 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:04:44.855676 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841644 5001 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:04:44.855676 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841646 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:04:44.855676 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841649 5001 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:04:44.855676 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841651 5001 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:04:44.855676 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841654 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:04:44.856351 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841657 5001 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:04:44.856351 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841659 5001 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:04:44.856351 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841662 5001 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:04:44.856351 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841665 5001 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:04:44.856351 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841667 5001 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:04:44.856351 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841671 5001 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:04:44.856351 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841673 5001 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:04:44.856351 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841676 5001 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:04:44.856351 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841679 5001 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:04:44.856351 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841683 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:04:44.856351 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841685 5001 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:04:44.856351 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841688 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:04:44.856351 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.841691 5001 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:04:44.856351 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.845752 5001 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 00:04:44.856351 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.845769 5001 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 00:04:44.856351 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.845788 5001 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 00:04:44.856351 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.845793 5001 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 00:04:44.856351 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845855 5001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:04:44.856351 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845860 5001 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:04:44.856351 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845864 5001 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:04:44.856853 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845867 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:04:44.856853 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845870 5001 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:04:44.856853 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845874 5001 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:04:44.856853 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845877 5001 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:04:44.856853 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845880 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:04:44.856853 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845883 5001 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:04:44.856853 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845885 5001 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:04:44.856853 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845888 5001 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:04:44.856853 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845891 5001 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:04:44.856853 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845893 5001 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:04:44.856853 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845896 5001 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:04:44.856853 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845899 5001 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:04:44.856853 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845901 5001 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:04:44.856853 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845904 5001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:04:44.856853 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845907 5001 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:04:44.856853 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845923 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:04:44.856853 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845926 5001 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:04:44.856853 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845928 5001 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:04:44.856853 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845933 5001 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:04:44.857324 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845937 5001 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:04:44.857324 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845940 5001 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:04:44.857324 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845943 5001 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:04:44.857324 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845946 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:04:44.857324 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845948 5001 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:04:44.857324 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845951 5001 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:04:44.857324 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845955 5001 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:04:44.857324 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845965 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:04:44.857324 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845969 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:04:44.857324 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845972 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:04:44.857324 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845975 5001 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:04:44.857324 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845978 5001 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:04:44.857324 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845980 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:04:44.857324 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845983 5001 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:04:44.857324 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845986 5001 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:04:44.857324 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845989 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:04:44.857324 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845992 5001 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:04:44.857324 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.845996 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:04:44.857324 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846001 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:04:44.857773 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846005 5001 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:04:44.857773 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846009 5001 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:04:44.857773 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846014 5001 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:04:44.857773 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846018 5001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:04:44.857773 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846022 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:04:44.857773 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846026 5001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:04:44.857773 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846030 5001 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:04:44.857773 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846034 5001 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:04:44.857773 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846039 5001 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:04:44.857773 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846042 5001 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:04:44.857773 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846045 5001 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:04:44.857773 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846047 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:04:44.857773 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846051 5001 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:04:44.857773 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846054 5001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:04:44.857773 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846057 5001 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:04:44.857773 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846060 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:04:44.857773 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846062 5001 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:04:44.857773 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846065 5001 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:04:44.857773 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846068 5001 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:04:44.857773 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846071 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:04:44.858475 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846073 5001 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:04:44.858475 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846082 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:04:44.858475 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846085 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:04:44.858475 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846088 5001 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:04:44.858475 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846091 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:04:44.858475 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846093 5001 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:04:44.858475 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846096 5001 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:04:44.858475 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846099 5001 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:04:44.858475 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846101 5001 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:04:44.858475 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846104 5001 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:04:44.858475 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846107 5001 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:04:44.858475 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846109 5001 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:04:44.858475 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846117 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:04:44.858475 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846120 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:04:44.858475 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846123 5001 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:04:44.858475 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846126 5001 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:04:44.858475 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846129 5001 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:04:44.858475 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846131 5001 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:04:44.858475 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846134 5001 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:04:44.858475 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846137 5001 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:04:44.858475 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846139 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:04:44.859045 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846142 5001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:04:44.859045 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846145 5001 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:04:44.859045 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846147 5001 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:04:44.859045 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846150 5001 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:04:44.859045 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846152 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:04:44.859045 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846155 5001 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:04:44.859045 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846157 5001 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:04:44.859045 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846160 5001 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:04:44.859045 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846163 5001 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:04:44.859045 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846166 5001 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:04:44.859045 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846168 5001 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:04:44.859045 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846171 5001 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:04:44.859045 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846173 5001 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:04:44.859045 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846182 5001 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:04:44.859045 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846185 5001 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:04:44.859045 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846187 5001 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:04:44.859045 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846190 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:04:44.859045 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846193 5001 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:04:44.859045 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846195 5001 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:04:44.859504 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846198 5001 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:04:44.859504 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846200 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:04:44.859504 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846203 5001 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:04:44.859504 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846205 5001 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:04:44.859504 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846208 5001 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:04:44.859504 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.846237 5001 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:04:44.859504 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846420 5001 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:04:44.859504 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846425 5001 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:04:44.859504 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846428 5001 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:04:44.859504 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846432 5001 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:04:44.859504 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846434 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:04:44.859504 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846437 5001 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:04:44.859504 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846440 5001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:04:44.859504 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846443 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:04:44.859504 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846445 5001 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:04:44.859860 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846448 5001 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:04:44.859860 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846451 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:04:44.859860 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846453 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:04:44.859860 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846456 5001 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:04:44.859860 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846459 5001 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:04:44.859860 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846461 5001 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:04:44.859860 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846464 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:04:44.859860 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846467 5001 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:04:44.859860 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846469 5001 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:04:44.859860 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846472 5001 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:04:44.859860 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846475 5001 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:04:44.859860 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846477 5001 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:04:44.859860 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846489 5001 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:04:44.859860 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846500 5001 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:04:44.859860 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846503 5001 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:04:44.859860 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846506 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:04:44.859860 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846508 5001 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:04:44.859860 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846511 5001 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:04:44.859860 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846514 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:04:44.860345 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846516 5001 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:04:44.860345 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846519 5001 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:04:44.860345 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846521 5001 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:04:44.860345 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846524 5001 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:04:44.860345 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846526 5001 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:04:44.860345 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846529 5001 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:04:44.860345 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846532 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:04:44.860345 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846534 5001 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:04:44.860345 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846537 5001 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:04:44.860345 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846539 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:04:44.860345 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846541 5001 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:04:44.860345 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846544 5001 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:04:44.860345 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846547 5001 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:04:44.860345 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846549 5001 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:04:44.860345 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846552 5001 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:04:44.860345 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846555 5001 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:04:44.860345 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846557 5001 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:04:44.860345 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846560 5001 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:04:44.860345 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846562 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:04:44.860345 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846565 5001 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:04:44.860818 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846568 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:04:44.860818 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846570 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:04:44.860818 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846573 5001 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:04:44.860818 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846575 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:04:44.860818 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846578 5001 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:04:44.860818 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846581 5001 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:04:44.860818 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846589 5001 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:04:44.860818 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846592 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:04:44.860818 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846595 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:04:44.860818 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846598 5001 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:04:44.860818 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846600 5001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:04:44.860818 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846603 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:04:44.860818 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846605 5001 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:04:44.860818 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846607 5001 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:04:44.860818 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846610 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:04:44.860818 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846612 5001 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:04:44.860818 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846615 5001 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:04:44.860818 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846618 5001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:04:44.860818 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846620 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:04:44.861379 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846623 5001 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:04:44.861379 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846625 5001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:04:44.861379 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846628 5001 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:04:44.861379 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846630 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:04:44.861379 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846633 5001 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:04:44.861379 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846653 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:04:44.861379 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846657 5001 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:04:44.861379 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846660 5001 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:04:44.861379 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846664 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:04:44.861379 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846667 5001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:04:44.861379 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846670 5001 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:04:44.861379 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846672 5001 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:04:44.861379 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846675 5001 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:04:44.861379 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846678 5001 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:04:44.861379 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846681 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:04:44.861379 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846685 5001 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:04:44.861379 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846689 5001 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:04:44.861379 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846692 5001 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:04:44.861379 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846695 5001 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:04:44.861834 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846698 5001 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:04:44.861834 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846706 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:04:44.861834 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846710 5001 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:04:44.861834 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846712 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:04:44.861834 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846715 5001 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:04:44.861834 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846718 5001 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:04:44.861834 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846721 5001 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:04:44.861834 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846723 5001 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:04:44.861834 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846726 5001 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:04:44.861834 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846728 5001 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:04:44.861834 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846731 5001 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:04:44.861834 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846734 5001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:04:44.861834 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846736 5001 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:04:44.861834 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846739 5001 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:04:44.861834 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846743 5001 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:04:44.861834 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846746 5001 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:04:44.861834 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846749 5001 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:04:44.861834 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846752 5001 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:04:44.861834 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846755 5001 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:04:44.861834 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846757 5001 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:04:44.862434 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.846781 5001 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:04:44.862434 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846825 5001 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:04:44.862434 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846829 5001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:04:44.862434 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846832 5001 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:04:44.862434 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846835 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:04:44.862434 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846837 5001 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:04:44.862434 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846840 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:04:44.862434 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846843 5001 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:04:44.862434 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846846 5001 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:04:44.862434 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846848 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:04:44.862434 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846851 5001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:04:44.862434 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846853 5001 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:04:44.862434 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846856 5001 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:04:44.862434 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846858 5001 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:04:44.862434 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846866 5001 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:04:44.862434 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846869 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:04:44.862881 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846873 5001 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:04:44.862881 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846878 5001 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:04:44.862881 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846881 5001 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:04:44.862881 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846884 5001 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:04:44.862881 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846886 5001 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:04:44.862881 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846889 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:04:44.862881 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846892 5001 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:04:44.862881 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846895 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:04:44.862881 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846897 5001 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:04:44.862881 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846900 5001 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:04:44.862881 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846903 5001 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:04:44.862881 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846905 5001 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:04:44.862881 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846919 5001 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:04:44.862881 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846922 5001 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:04:44.862881 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846924 5001 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:04:44.862881 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846927 5001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:04:44.862881 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846930 5001 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:04:44.862881 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846935 5001 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:04:44.862881 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846939 5001 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:04:44.863368 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846943 5001 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:04:44.863368 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846946 5001 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:04:44.863368 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846949 5001 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:04:44.863368 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846951 5001 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:04:44.863368 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846954 5001 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:04:44.863368 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846957 5001 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:04:44.863368 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846960 5001 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:04:44.863368 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846962 5001 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:04:44.863368 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846965 5001 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:04:44.863368 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846968 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:04:44.863368 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846970 5001 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:04:44.863368 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846973 5001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:04:44.863368 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846981 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:04:44.863368 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846985 5001 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:04:44.863368 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846987 5001 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:04:44.863368 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846991 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:04:44.863368 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846995 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:04:44.863368 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.846998 5001 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:04:44.863368 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847000 5001 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:04:44.863842 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847003 5001 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:04:44.863842 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847006 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:04:44.863842 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847008 5001 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:04:44.863842 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847011 5001 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:04:44.863842 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847014 5001 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:04:44.863842 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847016 5001 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:04:44.863842 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847019 5001 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:04:44.863842 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847022 5001 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:04:44.863842 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847024 5001 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:04:44.863842 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847027 5001 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:04:44.863842 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847030 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:04:44.863842 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847033 5001 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:04:44.863842 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847036 5001 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:04:44.863842 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847039 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:04:44.863842 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847041 5001 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:04:44.863842 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847044 5001 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:04:44.863842 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847046 5001 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:04:44.863842 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847049 5001 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:04:44.863842 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847052 5001 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:04:44.863842 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847054 5001 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:04:44.864397 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847057 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:04:44.864397 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847059 5001 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:04:44.864397 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847062 5001 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:04:44.864397 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847064 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:04:44.864397 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847067 5001 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:04:44.864397 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847069 5001 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:04:44.864397 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847078 5001 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:04:44.864397 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847083 5001 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:04:44.864397 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847087 5001 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:04:44.864397 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847094 5001 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:04:44.864397 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847106 5001 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:04:44.864397 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847111 5001 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:04:44.864397 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847114 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:04:44.864397 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847118 5001 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:04:44.864397 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847123 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:04:44.864397 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847127 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:04:44.864397 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847130 5001 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:04:44.864397 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847133 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:04:44.864397 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847136 5001 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:04:44.864397 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847138 5001 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:04:44.864886 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847141 5001 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:04:44.864886 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847144 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:04:44.864886 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847146 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:04:44.864886 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847150 5001 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:04:44.864886 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847153 5001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:04:44.864886 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847156 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:04:44.864886 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847158 5001 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:04:44.864886 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847161 5001 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:04:44.864886 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847164 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:04:44.864886 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847167 5001 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:04:44.864886 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847170 5001 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:04:44.864886 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847173 5001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:04:44.864886 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:44.847176 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:04:44.864886 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.847239 5001 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 00:04:44.864886 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.848250 5001 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 00:04:44.864886 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.848305 5001 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 00:04:44.864886 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.848496 5001 server.go:1014] "Starting client certificate rotation" Oct 30 00:04:44.864886 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.848721 5001 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 00:04:44.865330 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.849005 5001 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 23:59:04 +0000 UTC" deadline="2028-04-21 11:06:13.057061224 +0000 UTC" Oct 30 00:04:44.865330 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.849139 5001 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12947h1m28.207927053s" Oct 30 00:04:44.865330 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.850372 5001 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 00:04:44.865330 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.851412 5001 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 00:04:44.865330 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.852583 5001 log.go:25] "Validated CRI v1 runtime API" Oct 30 00:04:44.865330 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.854205 5001 log.go:25] "Validated CRI v1 image API" Oct 30 00:04:44.865330 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.854894 5001 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 00:04:44.865330 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.856784 5001 fs.go:135] Filesystem UUIDs: map[7827ee83-3ec9-436f-8d36-553a3f474bd7:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 90d22c60-f9c8-42ce-b35d-c0cf43c06df3:/dev/nvme0n1p4] Oct 30 00:04:44.865552 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.856804 5001 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/062d2ebaf412c97f991de7bda2e31d48a84793a3b68a7df39804ea228970a3cb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/062d2ebaf412c97f991de7bda2e31d48a84793a3b68a7df39804ea228970a3cb/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0962758c857bfd33f7077d1299e3caff3ae85f46d85bfa240cc0362d3a74fe0c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0962758c857bfd33f7077d1299e3caff3ae85f46d85bfa240cc0362d3a74fe0c/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/115ab9e9fecc7cf88524a0ecdc1649c7caec7db2b46b6428070d63a5fbf22474/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/115ab9e9fecc7cf88524a0ecdc1649c7caec7db2b46b6428070d63a5fbf22474/userdata/shm major:0 minor:324 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1f4f8d7cfd7163a3446b5059aa1b4fa1ee1529c9f2ad7703868530c29b5acbae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1f4f8d7cfd7163a3446b5059aa1b4fa1ee1529c9f2ad7703868530c29b5acbae/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2278a9b540030faf817efed78d6f518955498ae814e860c0c4a05446856e2b59/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2278a9b540030faf817efed78d6f518955498ae814e860c0c4a05446856e2b59/userdata/shm major:0 minor:49 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2a928774f503438c73c71ebc467269f535dce7bfd334c709327ea46cdfd13a9b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2a928774f503438c73c71ebc467269f535dce7bfd334c709327ea46cdfd13a9b/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2de8ec0b28a0011e709a226b0a443eb6a46110a90d420e78df8898b980728aed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2de8ec0b28a0011e709a226b0a443eb6a46110a90d420e78df8898b980728aed/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/43abae8136f399bc01fc7520685f1890ebb0d48a0060151959535418cf4d3f98/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/43abae8136f399bc01fc7520685f1890ebb0d48a0060151959535418cf4d3f98/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/45400f99a6a084f4caa35e5d1fa8e58afc39f1b891852ef408faf555bef06b64/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/45400f99a6a084f4caa35e5d1fa8e58afc39f1b891852ef408faf555bef06b64/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4d032dee89180d1a063f800c46cf9bee3a78b9a8137597408ad3e4e26624ebaf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4d032dee89180d1a063f800c46cf9bee3a78b9a8137597408ad3e4e26624ebaf/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5306018f5ceb7f93fc5e1cf1adc00fb3c87e0f81f729043a5983f1feb3fd0757/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5306018f5ceb7f93fc5e1cf1adc00fb3c87e0f81f729043a5983f1feb3fd0757/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/65987899f63eeb90b91417e14c7c9a09d3a46338421ca761818ad060f4639501/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/65987899f63eeb90b91417e14c7c9a09d3a46338421ca761818ad060f4639501/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/709f1626e391eae94355f8e5c2b187186a5c38c08846b951d629e2f53e8fa9c1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/709f1626e391eae94355f8e5c2b187186a5c38c08846b951d629e2f53e8fa9c1/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7347e4b08319778fdcd85e69581d07afc3de3b53fc4f15668b00d27490ee6793/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7347e4b08319778fdcd85e69581d07afc3de3b53fc4f15668b00d27490ee6793/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8710df781396f3314540cd1fdb27dfde39ee527d286c2e27699590bb90d4588d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8710df781396f3314540cd1fdb27dfde39ee527d286c2e27699590bb90d4588d/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c48de3e101e60c8a453a3f8a53c5c73ef1f5df30b21f5c8f2d45e4752e501bc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c48de3e101e60c8a453a3f8a53c5c73ef1f5df30b21f5c8f2d45e4752e501bc/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/938033fc66565d9fc00b418f0e9cf97a5d76e65f9284a69ab3cd167e0a482912/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/938033fc66565d9fc00b418f0e9cf97a5d76e65f9284a69ab3cd167e0a482912/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1ca01f6bedc6b25cc4de9362a8815a98eafbf74a13540072bca7da1aeefff30/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1ca01f6bedc6b25cc4de9362a8815a98eafbf74a13540072bca7da1aeefff30/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ba818f2ebc5b2cbeb9424de1c06d70910d2c2f9cc6094887b0ca46cef7294654/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ba818f2ebc5b2cbeb9424de1c06d70910d2c2f9cc6094887b0ca46cef7294654/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/baa7cbab439b2099259e9678bc017a1fb76d1b2165ad948b41c6994d2b387a38/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/baa7cbab439b2099259e9678bc017a1fb76d1b2165ad948b41c6994d2b387a38/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bbcafe6f8a6b21a56e4df1a6215e41026812e4fc5f2e8a3a816e0940661f3f49/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bbcafe6f8a6b21a56e4df1a6215e41026812e4fc5f2e8a3a816e0940661f3f49/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cae5f4674cc009c6bbbe513a11b55d9f505be0f44cf15567f37a9eaec203b977/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cae5f4674cc009c6bbbe513a11b55d9f505be0f44cf15567f37a9eaec203b977/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dbc06f0c0d480c5387e4cf6ad3b840b61fcc9f9eec5cd5ca71fb4f9d0e28835a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dbc06f0c0d480c5387e4cf6ad3b840b61fcc9f9eec5cd5ca71fb4f9d0e28835a/userdata/shm major:0 minor:337 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f7a8bc416f053eedd7084a1813fbc0d84ac2599e5f8b7717bd3405aa0eba9a9c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f7a8bc416f053eedd7084a1813fbc0d84ac2599e5f8b7717bd3405aa0eba9a9c/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0440635e-2cf5-4f40-a4f5-e716376044ba/volumes/kubernetes.io~projected/kube-api-access-7xgwg:{mountpoint:/var/lib/kubelet/pods/0440635e-2cf5-4f40-a4f5-e716376044ba/volumes/kubernetes.io~projected/kube-api-access-7xgwg major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09f5d7e2-51b8-4153-89c5-674d67065762/volumes/kubernetes.io~projected/kube-api-access-lgkdb:{mountpoint:/var/lib/kubelet/pods/09f5d7e2-51b8-4153-89c5-674d67065762/volumes/kubernetes.io~projected/kube-api-access-lgkdb major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fed8e88-bc49-4564-abcc-7d0fff068aa4/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/0fed8e88-bc49-4564-abcc-7d0fff068aa4/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fed8e88-bc49-4564-abcc-7d0fff068aa4/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/0fed8e88-bc49-4564-abcc-7d0fff068aa4/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fed8e88-bc49-4564-abcc-7d0fff068aa4/volumes/kubernetes.io~projected/kube-api-access-qxvlt:{mountpoint:/var/lib/kubelet/pods/0fed8e88-bc49-4564-abcc-7d0fff068aa4/volumes/kubernetes.io~projected/kube-api-access-qxvlt major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10514d09-1dbf-47c2-bc1b-54a15f384cb3/volumes/kubernetes.io~projected/kube-api-access-h6vsb:{mountpoint:/var/lib/kubelet/pods/10514d09-1dbf-47c2-bc1b-54a15f384cb3/volumes/kubernetes.io~projected/kube-api-access-h6vsb major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12aceb3a-2476-4efc-94d8-e6ec78cfc680/volumes/kubernetes.io~projected/kube-api-access-9vrjg:{mountpoint:/var/lib/kubelet/pods/12aceb3a-2476-4efc-94d8-e6ec78cfc680/volumes/kubernetes.io~projected/kube-api-access-9vrjg major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18a4ebbe-c390-49d8-99e7-858bfaa3b834/volumes/kubernetes.io~projected/kube-api-access-s8xpk:{mountpoint:/var/lib/kubelet/pods/18a4ebbe-c390-49d8-99e7-858bfaa3b834/volumes/kubernetes.io~projected/kube-api-access-s8xpk major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2277c280-7906-4c42-87f1-7941012a2d31/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/2277c280-7906-4c42-87f1-7941012a2d31/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2277c280-7906-4c42-87f1-7941012a2d31/volumes/kubernetes.io~projected/kube-api-access-l48jx:{mountpoint:/var/lib/kubelet/pods/2277c280-7906-4c42-87f1-7941012a2d31/volumes/kubernetes.io~projected/kube-api-access-l48jx major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2277c280-7906-4c42-87f1-7941012a2d31/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/2277c280-7906-4c42-87f1-7941012a2d31/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27614fa1-66da-4d21-a0d1-0ac85a9bdd13/volumes/kubernetes.io~projected/kube-api-access-wmw7p:{mountpoint:/var/lib/kubelet/pods/27614fa1-66da-4d21-a0d1-0ac85a9bdd13/volumes/kubernetes.io~projected/kube-api-access-wmw7p major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30673bf9-d332-4592-afce-b16db49876cb/volumes/kubernetes.io~projected/kube-api-access-skw5p:{mountpoint:/var/lib/kubelet/pods/30673bf9-d332-4592-afce-b16db49876cb/volumes/kubernetes.io~projected/kube-api-access-skw5p major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31c2770e-628c-4bcd-a088-73f190c952cd/volumes/kubernetes.io~projected/kube-api-access-s2dzh:{mountpoint:/var/lib/kubelet/pods/31c2770e-628c-4bcd-a088-73f190c952cd/volumes/kubernetes.io~projected/kube-api-access-s2dzh major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32309642-7cba-4361-83b3-4680659c0544/volumes/kubernetes.io~projected/kube-api-access-5dpbj:{mountpoint:/var/lib/kubelet/pods/32309642-7cba-4361-83b3-4680659c0544/volumes/kubernetes.io~projected/kube-api-access-5dpbj major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8/volumes/kubernetes.io~projected/kube-api-access-6xrgq:{mountpoint:/var/lib/kubelet/pods/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8/volumes/kubernetes.io~projected/kube-api-access-6xrgq major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4350294b-64f3-4efd-9619-90c013d82432/volumes/kubernetes.io~projected/kube-api-access-7jzxf:{mountpoint:/var/lib/kubelet/pods/4350294b-64f3-4efd-9619-90c013d82432/volumes/kubernetes.io~projected/kube-api-access-7jzxf major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5166c4d7-2b82-4bbd-8d58-836cb274ed25/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/5166c4d7-2b82-4bbd-8d58-836cb274ed25/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/526cc561-1bbd-4079-8a2f-326f939753e0/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/526cc561-1bbd-4079-8a2f-326f939753e0/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/526cc561-1bbd-4079-8a2f-326f939753e0/volumes/kubernetes.io~projected/kube-api-access-cngbs:{mountpoint:/var/lib/kubelet/pods/526cc561-1bbd-4079-8a2f-326f939753e0/volumes/kubernetes.io~projected/kube-api-access-cngbs major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/526cc561-1bbd-4079-8a2f-326f939753e0/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/526cc561-1bbd-4079-8a2f-326f939753e0/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/526cc561-1bbd-4079-8a2f-326f939753e0/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/526cc561-1bbd-4079-8a2f-326f939753e0/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f2b3396-a95d-4bc7-902f-1daf3dd689fd/volumes/kubernetes.io~projected/kube-api-access-ddgh4:{mountpoint:/var/lib/kubelet/pods/6f2b3396-a95d-4bc7-902f-1daf3dd689fd/volumes/kubernetes.io~projected/kube-api-access-ddgh4 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85b08631-ff08-461f-8bda-6a64a609e1f1/volumes/kubernetes.io~projected/kube-api-access-4g6m6:{mountpoint:/var/lib/kubelet/pods/85b08631-ff08-461f-8bda-6a64a609e1f1/volumes/kubernetes.io~projected/kube-api-access-4g6m6 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a82abd0-5057-4361-9b0d-7a4f23ae97d1/volumes/kubernetes.io~projected/kube-api-access-xd5wb:{mountpoint:/var/lib/kubelet/pods/8a82abd0-5057-4361-9b0d-7a4f23ae97d1/volumes/kubernetes.io~projected/kube-api-access-xd5wb major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a82abd0-5057-4361-9b0d-7a4f23ae97d1/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/8a82abd0-5057-4361-9b0d-7a4f23ae97d1/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d95c49a-0861-4878-b10f-6b51c391b19e/volumes/kubernetes.io~projected/kube-api-access-dzmtr:{mountpoint:/var/lib/kubelet/pods/8d95c49a-0861-4878-b10f-6b51c391b19e/volumes/kubernetes.io~projected/kube-api-access-dzmtr major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d080b8e-cca6-4040-830e-cc9e9e25c50a/volumes/kubernetes.io~projected/kube-api-access-6ht68:{mountpoint:/var/lib/kubelet/pods/9d080b8e-cca6-4040-830e-cc9e9e25c50a/volumes/kubernetes.io~projected/kube-api-access-6ht68 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d080b8e-cca6-4040-830e-cc9e9e25c50a/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/9d080b8e-cca6-4040-830e-cc9e9e25c50a/volumes/kubernetes.io~secret/default-certificate major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d080b8e-cca6-4040-830e-cc9e9e25c50a/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/9d080b8e-cca6-4040-830e-cc9e9e25c50a/volumes/kubernetes.io~secret/stats-auth major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9fa688d8-29ca-4c1f-926a-f1a4099e6b44/volumes/kubernetes.io~projected/kube-api-access-6cv5p:{mountpoint:/var/lib/kubelet/pods/9fa688d8-29ca-4c1f-926a-f1a4099e6b44/volumes/kubernetes.io~projected/kube-api-access-6cv5p major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9fa688d8-29ca-4c1f-926a-f1a4099e6b44/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/9fa688d8-29ca-4c1f-926a-f1a4099e6b44/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a20efc24-e4cb-4075-911e-bfc43dcfc20b/volumes/kubernetes.io~projected/kube-api-access-dpg2j:{mountpoint:/var/lib/kubelet/pods/a20efc24-e4cb-4075-911e-bfc43dcfc20b/volumes/kubernetes.io~projected/kube-api-access-dpg2j major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9e7a0c8-43b3-4f8c-8870-505edace7f78/volumes/kubernetes.io~projected/kube-api-access-cs94l:{mountpoint:/var/lib/kubelet/pods/a9e7a0c8-43b3-4f8c-8870-505edace7f78/volumes/kubernetes.io~projected/kube-api-access-cs94l major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/acc82f37-fad2-424d-89d0-5e5876cb75de/volumes/kubernetes.io~projected/kube-api-access-kfc4x:{mountpoint:/var/lib/kubelet/pods/acc82f37-fad2-424d-89d0-5e5876cb75de/volumes/kubernetes.io~projected/kube-api-access-kfc4x major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aff3a44b-6cad-449e-8733-f85e9899ecc3/volumes/kubernetes.io~projected/kube-api-access-56j8l:{mountpoint:/var/lib/kubelet/pods/aff3a44b-6cad-449e-8733-f85e9899ecc3/volumes/kubernetes.io~projected/kube-api-access-56j8l major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1a647f1-e715-4a8a-8b44-0196b3b1af27/volumes/kubernetes.io~projected/kube-api-access-2v899:{mountpoint:/var/lib/kubelet/pods/c1a647f1-e715-4a8a-8b44-0196b3b1af27/volumes/kubernetes.io~projected/kube-api-access-2v899 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1a647f1-e715-4a8a-8b44-0196b3b1af27/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/c1a647f1-e715-4a8a-8b44-0196b3b1af27/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c3ce5e93-7d27-4fa6-b360-e571d8f18481/volumes/kubernetes.io~projected/kube-api-access-v4snt:{mountpoint:/var/lib/kubelet/pods/c3ce5e93-7d27-4fa6-b360-e571d8f18481/volumes/kubernetes.io~projected/kube-api-access-v4snt major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2413398-197b-4cad-8dc4-9bba9effe3b7/volumes/kubernetes.io~projected/kube-api-access-spcsf:{mountpoint:/var/lib/kubelet/pods/d2413398-197b-4cad-8dc4-9bba9effe3b7/volumes/kubernetes.io~projected/kube-api-access-spcsf major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de19ca9c-b22d-49a4-a5a6-a5f3a61bf892/volumes/kubernetes.io~projected/kube-api-access-mglpv:{mountpoint:/var/lib/kubelet/pods/de19ca9c-b22d-49a4-a5a6-a5f3a61bf892/volumes/kubernetes.io~projected/kube-api-access-mglpv major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de19ca9c-b22d-49a4-a5a6-a5f3a61bf892/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/de19ca9c-b22d-49a4-a5a6-a5f3a61bf892/volumes/kubernetes.io~secret/cert major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb8917db-c1ba-43af-995e-1948de69aa29/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/fb8917db-c1ba-43af-995e-1948de69aa29/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd8f9a4a-5b77-4a52-9ea9-a12245c00489/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/fd8f9a4a-5b77-4a52-9ea9-a12245c00489/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd8f9a4a-5b77-4a52-9ea9-a12245c00489/volumes/kubernetes.io~projected/kube-api-access-f87wl:{mountpoint:/var/lib/kubelet/pods/fd8f9a4a-5b77-4a52-9ea9-a12245c00489/volumes/kubernetes.io~projected/kube-api-access-f87wl major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd8f9a4a-5b77-4a52-9ea9-a12245c00489/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/fd8f9a4a-5b77-4a52-9ea9-a12245c00489/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd8f9a4a-5b77-4a52-9ea9-a12245c00489/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/fd8f9a4a-5b77-4a52-9ea9-a12245c00489/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:210 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/02df3fbe4455684293616d1fdd3375758bb667b31155a7962063d9cf233c12a0/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/aa17cdd50793821a2de8d7ba602bada922f4a79cb1245e41de637577dba85303/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/3e094070d7f04721e9d7a4843ab4b256409071658d5c6efb592a0ffd969a4a77/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/61c098cf37a9af925d5ba4b330b37deb80d7c01a560331c8b76db78ada5825fe/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/b4525c036ee8af205f128b6aed8657394e8bcee0310f021f86caab0834ef788d/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/b9878dd667f2193a12981fa54bbad87fe2f9dc252ddc118b92131b19c5214971/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/563065a72b32adce35546020318faa9f1e345bc3de27f8a9f7749c8c9abf2b9f/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/0ec4a8360e4fd59918a6513d95a46b6ceeee62251d3020185cfc90a2d0b139a2/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/24e4260439b4653ec538098aa163570f254e5e89ec560fa5d7e7ee99fd619542/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/0f394156e9bc49a7164a02578c23b3921749a55b52774ac31980acb3159d6f4b/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/e1282bdd7d231547a7f92271b8204ea4089daac7113f96605cf1de0ec3504b77/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/b3405d86768f041480c3837093f75947348f2516ca0c266a1bbe08db321779e9/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/d7dab81fc091d6a4cbd7af58cf7a93de09951033d715e61a6884a274cad470d2/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/7e9a0867febbdab7640aea3a765ad65f17efc8b110a0b7bc384befdb362cc76e/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/524ec394c95acc92977f1046f44ededb155f981b09f1f9e321e86f9086e43a7d/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/a7a80943dc3f2605d615a439150b1bf3b998303352f9eabc22882205a80e5e0e/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/fc45ce368823bb0a2f541f25d8d1842e0b35f6b07b7de09b61fbdb4ee46ee15b/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/c182d8684ea77e37f605082a7419daac05e2013174bd234c4a30e3eff4581ea0/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/008b76a3b19f69092e7aac312623962289c9566e2e6f7b6945a5d6c18defe77d/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/352bd10e0bdec83da17fcebf4e9d21af67938bfe0627d5239d9f890e03b89bb7/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/701299ad2188369e64b6bb8ec72c46cac2047b4540cf86cb48176157d85af6e8/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/f0b5399cdbb27b2b066191dc8e393be29994c93ab957f32d3fc9f6f17a0ffb60/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/8ddf2c8d4242b16f0a7ed3cddccc19e4d64d4e53189fca12cba6943aecfab04a/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/d86d2d13adb9bdea54d3e8e9e9c343f4962db1e1e862040778ceb5485eb57361/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/795dc2b9d715d7cf5f1e82882be713e2e0e3e58334562790bcc2adc32994c5c8/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/b1e707b3a9792da8f255030c6e7e4f4ce07c3d8fec5d668f027ec73bf3da6eee/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/afdaa126c422f2420665aacc609c342e96bf7dc2e9bc8c8dcaf8078a036dcfee/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/b56c305679db6b8ac8bb431c05ba5f85ead0124cd6733c8bfbdc02d2b26f4987/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/c0794ea40492340f955e161913cb400686f86174a313452793e0a49c8ffc52f2/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/25da23ce11c13313a9e694e7dd8b0d9360fb5ec437f58b4097891f8b1c43c6fb/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/97abdd9c1097ec3083f0445aafc50909f3a40ca57c0f1960cb3ad74e72c46fe3/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/2eb711a14a8d143ccc4588ea5b23e9a0cfcec1b8f02555105a44eeab08b5fa50/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/c2f043389e21985eed3d7acc1f6b5eb81ba6c5a46c4076215aa999e3c419f71a/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/e2a89b78dbc87680673db611e5671dd912c0f01955189497ec76c2a2eb5e975d/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/c9dae8e9d6181fee676f4d5e267fd44b38d2c2c5ca450484b44626fb6a66607e/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/7d45f8b25e3ca2202d17380109b776d6a80cb37f4d37d892229e739c1099a279/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/0e7945c92b3e8ac948ff1511c4967a89f29e1358c4fd51c2817b6dc1a96865ca/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/ab645ec9051e9583e9e2705a186ed8594659470fb79ac917245bd31b3d6452b3/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/d7fc2ea1056b12b90122b267523ddde2aa87802f2bab4190a1b7b84a83c99580/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/8d04e822f133016ab6e2cf867c334c5a3450f2ddb00fbeecd8a80bab3276843e/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/06da049cb4db833734e8dd37c3ed05b09245ba60c2c626015ba62328ecd78cd1/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/b34de021a01cb1eba23cb58a92608c6708c5a2a5eb5e920157a7f7b12c39ca07/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/585d23f75a375070fd7f67e6ffb9fc0c492d5c20f161810164ccb5844e759f0c/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/bcf68b32784b90cb18ce10c74a7fb4f7846d26a62be59daa899c0e2bcdc15fa4/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/9dcb2938b900b302421f09c7b3001e59908f8955cb0ff15ac6cd5bda510b8480/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/fc4b8e423786e91f2c1858a8d7150878295e41f9f9c2e3a463fd0bc0e8a1919b/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/55e2fb2cf323e2987a193a6a67e720e74e58dcb2be54f81899659db4f1bd901f/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/19ac5472e6d11a942c08edf100415305e56a861ebe5a638fee9e704fdf0b796c/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-50:{mountpoint:/var/lib/containers/storage/overlay/5bfb1e22b76482ec9d3336333bdfc90232e2159542663a1e14062de657828128/merged major:0 minor:50 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/2e65d5e1017e435138784867f296b33f5e43a7592f3a0d3a6aef7e6fa6e69897/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/491363145e0bf94f03ab3aac1ae4beb02455db8cb47a8aee3cb2b361fd7c40db/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/1f3794efef5c7b6fbdbb3420a8c3f1967244eec273ebba7a7f28ad52f8f363e5/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/e5cb1f50342b943c63ae64c2590253970d8969fd9a4f134cbfce5371de3257a1/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/e056504ebbd8e0642c8ff24713f65b2b802d318000557f2aa3d09898756b30c5/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/23a4b4232f8dddc7ba4bf755d2add32d1eeae6e3c93dc554695a1c113e1b1d45/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/d07a0d385a2f1034b021fe038c48e11998016860aabe41ce07434b91474490be/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/77690f6bbb8d59cd8911bb2bf72c6a9cbf4724ae6c606287a082240fdbd9964a/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/289e697d546759c4acd8b06c62abe85380c0ee53d0e900ea1f072f5af81bf734/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 00:04:44.865877 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.865091 5001 manager.go:217] Machine: {Timestamp:2026-10-30 00:04:44.864614735 +0000 UTC m=+0.070115907 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095248 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28a533568465876b15a59ff6315a42 SystemUUID:ec28a533-5684-6587-6b15-a59ff6315a42 BootID:d8274494-8ca6-4261-b14d-df995affed70 Filesystems:[{Device:/run/containers/storage/overlay-containers/4d032dee89180d1a063f800c46cf9bee3a78b9a8137597408ad3e4e26624ebaf/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a82abd0-5057-4361-9b0d-7a4f23ae97d1/volumes/kubernetes.io~projected/kube-api-access-xd5wb DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cae5f4674cc009c6bbbe513a11b55d9f505be0f44cf15567f37a9eaec203b977/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7347e4b08319778fdcd85e69581d07afc3de3b53fc4f15668b00d27490ee6793/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8/volumes/kubernetes.io~projected/kube-api-access-6xrgq DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0fed8e88-bc49-4564-abcc-7d0fff068aa4/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de19ca9c-b22d-49a4-a5a6-a5f3a61bf892/volumes/kubernetes.io~projected/kube-api-access-mglpv DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/062d2ebaf412c97f991de7bda2e31d48a84793a3b68a7df39804ea228970a3cb/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fd8f9a4a-5b77-4a52-9ea9-a12245c00489/volumes/kubernetes.io~projected/kube-api-access-f87wl DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a20efc24-e4cb-4075-911e-bfc43dcfc20b/volumes/kubernetes.io~projected/kube-api-access-dpg2j DeviceMajor:0 DeviceMinor:288 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0fed8e88-bc49-4564-abcc-7d0fff068aa4/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/45400f99a6a084f4caa35e5d1fa8e58afc39f1b891852ef408faf555bef06b64/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8d95c49a-0861-4878-b10f-6b51c391b19e/volumes/kubernetes.io~projected/kube-api-access-dzmtr DeviceMajor:0 DeviceMinor:332 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/10514d09-1dbf-47c2-bc1b-54a15f384cb3/volumes/kubernetes.io~projected/kube-api-access-h6vsb DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2de8ec0b28a0011e709a226b0a443eb6a46110a90d420e78df8898b980728aed/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/27614fa1-66da-4d21-a0d1-0ac85a9bdd13/volumes/kubernetes.io~projected/kube-api-access-wmw7p DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de19ca9c-b22d-49a4-a5a6-a5f3a61bf892/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2277c280-7906-4c42-87f1-7941012a2d31/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/526cc561-1bbd-4079-8a2f-326f939753e0/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fd8f9a4a-5b77-4a52-9ea9-a12245c00489/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1f4f8d7cfd7163a3446b5059aa1b4fa1ee1529c9f2ad7703868530c29b5acbae/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5306018f5ceb7f93fc5e1cf1adc00fb3c87e0f81f729043a5983f1feb3fd0757/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d2413398-197b-4cad-8dc4-9bba9effe3b7/volumes/kubernetes.io~projected/kube-api-access-spcsf DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4350294b-64f3-4efd-9619-90c013d82432/volumes/kubernetes.io~projected/kube-api-access-7jzxf DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/32309642-7cba-4361-83b3-4680659c0544/volumes/kubernetes.io~projected/kube-api-access-5dpbj DeviceMajor:0 DeviceMinor:228 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/30673bf9-d332-4592-afce-b16db49876cb/volumes/kubernetes.io~projected/kube-api-access-skw5p DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0962758c857bfd33f7077d1299e3caff3ae85f46d85bfa240cc0362d3a74fe0c/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a9e7a0c8-43b3-4f8c-8870-505edace7f78/volumes/kubernetes.io~projected/kube-api-access-cs94l DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/baa7cbab439b2099259e9678bc017a1fb76d1b2165ad948b41c6994d2b387a38/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6f2b3396-a95d-4bc7-902f-1daf3dd689fd/volumes/kubernetes.io~projected/kube-api-access-ddgh4 DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9fa688d8-29ca-4c1f-926a-f1a4099e6b44/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fd8f9a4a-5b77-4a52-9ea9-a12245c00489/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fd8f9a4a-5b77-4a52-9ea9-a12245c00489/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f7a8bc416f053eedd7084a1813fbc0d84ac2599e5f8b7717bd3405aa0eba9a9c/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/12aceb3a-2476-4efc-94d8-e6ec78cfc680/volumes/kubernetes.io~projected/kube-api-access-9vrjg DeviceMajor:0 DeviceMinor:316 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2278a9b540030faf817efed78d6f518955498ae814e860c0c4a05446856e2b59/userdata/shm DeviceMajor:0 DeviceMinor:49 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/85b08631-ff08-461f-8bda-6a64a609e1f1/volumes/kubernetes.io~projected/kube-api-access-4g6m6 DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0fed8e88-bc49-4564-abcc-7d0fff068aa4/volumes/kubernetes.io~projected/kube-api-access-qxvlt DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9d080b8e-cca6-4040-830e-cc9e9e25c50a/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9d080b8e-cca6-4040-830e-cc9e9e25c50a/volumes/kubernetes.io~projected/kube-api-access-6ht68 DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/526cc561-1bbd-4079-8a2f-326f939753e0/volumes/kubernetes.io~projected/kube-api-access-cngbs DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/115ab9e9fecc7cf88524a0ecdc1649c7caec7db2b46b6428070d63a5fbf22474/userdata/shm DeviceMajor:0 DeviceMinor:324 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/43abae8136f399bc01fc7520685f1890ebb0d48a0060151959535418cf4d3f98/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/31c2770e-628c-4bcd-a088-73f190c952cd/volumes/kubernetes.io~projected/kube-api-access-s2dzh DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8710df781396f3314540cd1fdb27dfde39ee527d286c2e27699590bb90d4588d/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/65987899f63eeb90b91417e14c7c9a09d3a46338421ca761818ad060f4639501/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c48de3e101e60c8a453a3f8a53c5c73ef1f5df30b21f5c8f2d45e4752e501bc/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1ca01f6bedc6b25cc4de9362a8815a98eafbf74a13540072bca7da1aeefff30/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2277c280-7906-4c42-87f1-7941012a2d31/volumes/kubernetes.io~projected/kube-api-access-l48jx DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bbcafe6f8a6b21a56e4df1a6215e41026812e4fc5f2e8a3a816e0940661f3f49/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/938033fc66565d9fc00b418f0e9cf97a5d76e65f9284a69ab3cd167e0a482912/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/09f5d7e2-51b8-4153-89c5-674d67065762/volumes/kubernetes.io~projected/kube-api-access-lgkdb DeviceMajor:0 DeviceMinor:323 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/526cc561-1bbd-4079-8a2f-326f939753e0/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5166c4d7-2b82-4bbd-8d58-836cb274ed25/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2277c280-7906-4c42-87f1-7941012a2d31/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2a928774f503438c73c71ebc467269f535dce7bfd334c709327ea46cdfd13a9b/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/acc82f37-fad2-424d-89d0-5e5876cb75de/volumes/kubernetes.io~projected/kube-api-access-kfc4x DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0440635e-2cf5-4f40-a4f5-e716376044ba/volumes/kubernetes.io~projected/kube-api-access-7xgwg DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ba818f2ebc5b2cbeb9424de1c06d70910d2c2f9cc6094887b0ca46cef7294654/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c1a647f1-e715-4a8a-8b44-0196b3b1af27/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aff3a44b-6cad-449e-8733-f85e9899ecc3/volumes/kubernetes.io~projected/kube-api-access-56j8l DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d080b8e-cca6-4040-830e-cc9e9e25c50a/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/526cc561-1bbd-4079-8a2f-326f939753e0/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9fa688d8-29ca-4c1f-926a-f1a4099e6b44/volumes/kubernetes.io~projected/kube-api-access-6cv5p DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fb8917db-c1ba-43af-995e-1948de69aa29/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:293 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-50 DeviceMajor:0 DeviceMinor:50 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a82abd0-5057-4361-9b0d-7a4f23ae97d1/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/709f1626e391eae94355f8e5c2b187186a5c38c08846b951d629e2f53e8fa9c1/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c1a647f1-e715-4a8a-8b44-0196b3b1af27/volumes/kubernetes.io~projected/kube-api-access-2v899 DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/18a4ebbe-c390-49d8-99e7-858bfaa3b834/volumes/kubernetes.io~projected/kube-api-access-s8xpk DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c3ce5e93-7d27-4fa6-b360-e571d8f18481/volumes/kubernetes.io~projected/kube-api-access-v4snt DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dbc06f0c0d480c5387e4cf6ad3b840b61fcc9f9eec5cd5ca71fb4f9d0e28835a/userdata/shm DeviceMajor:0 DeviceMinor:337 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:062d2ebaf412c97 MacAddress:22:4d:07:4b:5e:b6 Speed:10000 Mtu:8901} {Name:115ab9e9fecc7cf MacAddress:86:b5:41:07:c0:ab Speed:10000 Mtu:8901} {Name:1f4f8d7cfd7163a MacAddress:4e:4a:60:9b:0f:f5 Speed:10000 Mtu:8901} {Name:2a928774f503438 MacAddress:e2:26:2a:45:7b:75 Speed:10000 Mtu:8901} {Name:2de8ec0b28a0011 MacAddress:92:c7:fb:dd:36:13 Speed:10000 Mtu:8901} {Name:45400f99a6a084f MacAddress:b6:5f:41:f7:79:5f Speed:10000 Mtu:8901} {Name:7347e4b08319778 MacAddress:92:77:4b:5f:1c:d4 Speed:10000 Mtu:8901} {Name:8710df781396f33 MacAddress:76:0a:11:46:9c:d5 Speed:10000 Mtu:8901} {Name:8c48de3e101e60c MacAddress:da:60:61:29:15:79 Speed:10000 Mtu:8901} {Name:938033fc66565d9 MacAddress:a6:98:2e:66:1b:2d Speed:10000 Mtu:8901} {Name:bbcafe6f8a6b21a MacAddress:6a:2d:56:0a:32:36 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c8:da:52:91 Speed:0 Mtu:9001} {Name:br-int MacAddress:4a:52:e0:e0:ff:a4 Speed:0 Mtu:8901} {Name:cae5f4674cc009c MacAddress:e2:b9:f1:b7:ad:2c Speed:10000 Mtu:8901} {Name:dbc06f0c0d480c5 MacAddress:26:73:e9:b1:c0:12 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c8:da:52:91 Speed:0 Mtu:9001} {Name:f7a8bc416f053ee MacAddress:3a:17:d6:26:05:5e Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:ee:01:6e:b6:ae:7c Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:a6:2c:ca:27:06:b0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 00:04:44.865877 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.865432 5001 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 00:04:44.865877 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.865476 5001 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 00:04:44.866269 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.866248 5001 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 00:04:44.866479 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.866271 5001 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-120.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 00:04:44.866564 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.866481 5001 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 00:04:44.866564 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.866497 5001 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 00:04:44.866564 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.866512 5001 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 00:04:44.866564 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.866529 5001 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 00:04:44.866564 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.866554 5001 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 00:04:44.867071 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.866792 5001 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 00:04:44.867071 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.867038 5001 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 00:04:44.867216 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.867104 5001 kubelet.go:493] "Attempting to sync node with API server" Oct 30 00:04:44.867216 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.867123 5001 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 00:04:44.867306 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.867233 5001 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 00:04:44.867354 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.867305 5001 kubelet.go:405] "Adding apiserver pod source" Oct 30 00:04:44.867393 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.867358 5001 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 00:04:44.868586 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.868569 5001 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 00:04:44.868586 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.868584 5001 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 00:04:44.869046 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.869026 5001 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 00:04:44.871330 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.871314 5001 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 00:04:44.871330 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.871329 5001 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 00:04:44.871451 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.871428 5001 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 00:04:44.871617 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.871606 5001 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 00:04:44.871648 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.871635 5001 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 00:04:44.871730 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.871721 5001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 00:04:44.871769 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.871735 5001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 00:04:44.871769 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.871742 5001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 00:04:44.871769 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.871747 5001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 00:04:44.871769 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.871753 5001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 00:04:44.871769 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.871759 5001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 00:04:44.871769 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.871765 5001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 00:04:44.871769 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.871771 5001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 00:04:44.872066 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.871780 5001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 00:04:44.872066 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.871786 5001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 00:04:44.872066 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.871796 5001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 00:04:44.872066 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.871805 5001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 00:04:44.872066 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.871828 5001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 00:04:44.872066 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.871834 5001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 00:04:44.875270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.875256 5001 server.go:1263] "Started kubelet" Oct 30 00:04:44.875344 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.875318 5001 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 00:04:44.875415 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.875356 5001 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 00:04:44.875474 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.875443 5001 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 00:04:44.875793 ip-10-0-143-120 systemd[1]: Started Kubernetes Kubelet. Oct 30 00:04:44.876651 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.876633 5001 server.go:317] "Adding debug handlers to kubelet server" Oct 30 00:04:44.876884 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.876683 5001 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 00:04:44.879884 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.879585 5001 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 00:04:44.880053 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.880035 5001 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 00:04:44.880142 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.880055 5001 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 00:04:44.880216 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.880187 5001 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 00:04:44.880453 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:44.880418 5001 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-120.ec2.internal\" not found" Oct 30 00:04:44.880453 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.880275 5001 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 00:04:44.880601 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.880501 5001 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 23:59:06 +0000 UTC" deadline="2028-04-11 00:22:56.997132746 +0000 UTC" Oct 30 00:04:44.880601 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.880536 5001 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12696h18m12.116600561s" Oct 30 00:04:44.881492 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.881382 5001 factory.go:55] Registering systemd factory Oct 30 00:04:44.881492 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.881410 5001 factory.go:223] Registration of the systemd container factory successfully Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.882400 5001 factory.go:153] Registering CRI-O factory Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.882420 5001 factory.go:223] Registration of the crio container factory successfully Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.882479 5001 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.882509 5001 factory.go:103] Registering Raw factory Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.882526 5001 manager.go:1199] Started watching for new ooms in manager Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.882882 5001 manager.go:319] Starting recovery of all containers Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.883981 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fa688d8-29ca-4c1f-926a-f1a4099e6b44" volumeName="kubernetes.io/configmap/9fa688d8-29ca-4c1f-926a-f1a4099e6b44-trusted-ca" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884031 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9e7a0c8-43b3-4f8c-8870-505edace7f78" volumeName="kubernetes.io/projected/a9e7a0c8-43b3-4f8c-8870-505edace7f78-kube-api-access-cs94l" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884045 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acc82f37-fad2-424d-89d0-5e5876cb75de" volumeName="kubernetes.io/projected/acc82f37-fad2-424d-89d0-5e5876cb75de-kube-api-access-kfc4x" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884058 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1a647f1-e715-4a8a-8b44-0196b3b1af27" volumeName="kubernetes.io/projected/c1a647f1-e715-4a8a-8b44-0196b3b1af27-kube-api-access-2v899" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884072 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="526cc561-1bbd-4079-8a2f-326f939753e0" volumeName="kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-kube-api-access-cngbs" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884084 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fa688d8-29ca-4c1f-926a-f1a4099e6b44" volumeName="kubernetes.io/configmap/9fa688d8-29ca-4c1f-926a-f1a4099e6b44-config" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884095 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09f5d7e2-51b8-4153-89c5-674d67065762" volumeName="kubernetes.io/configmap/09f5d7e2-51b8-4153-89c5-674d67065762-trusted-ca-bundle" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884106 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12aceb3a-2476-4efc-94d8-e6ec78cfc680" volumeName="kubernetes.io/projected/12aceb3a-2476-4efc-94d8-e6ec78cfc680-kube-api-access-9vrjg" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884119 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f2b3396-a95d-4bc7-902f-1daf3dd689fd" volumeName="kubernetes.io/projected/6f2b3396-a95d-4bc7-902f-1daf3dd689fd-kube-api-access-ddgh4" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884130 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a82abd0-5057-4361-9b0d-7a4f23ae97d1" volumeName="kubernetes.io/configmap/8a82abd0-5057-4361-9b0d-7a4f23ae97d1-config" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884142 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aff3a44b-6cad-449e-8733-f85e9899ecc3" volumeName="kubernetes.io/projected/aff3a44b-6cad-449e-8733-f85e9899ecc3-kube-api-access-56j8l" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884164 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1a647f1-e715-4a8a-8b44-0196b3b1af27" volumeName="kubernetes.io/configmap/c1a647f1-e715-4a8a-8b44-0196b3b1af27-config" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884177 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1a647f1-e715-4a8a-8b44-0196b3b1af27" volumeName="kubernetes.io/secret/c1a647f1-e715-4a8a-8b44-0196b3b1af27-serving-cert" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884189 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd8f9a4a-5b77-4a52-9ea9-a12245c00489" volumeName="kubernetes.io/secret/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-installation-pull-secrets" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884205 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0440635e-2cf5-4f40-a4f5-e716376044ba" volumeName="kubernetes.io/configmap/0440635e-2cf5-4f40-a4f5-e716376044ba-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884218 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fed8e88-bc49-4564-abcc-7d0fff068aa4" volumeName="kubernetes.io/projected/0fed8e88-bc49-4564-abcc-7d0fff068aa4-kube-api-access-qxvlt" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884233 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10514d09-1dbf-47c2-bc1b-54a15f384cb3" volumeName="kubernetes.io/configmap/10514d09-1dbf-47c2-bc1b-54a15f384cb3-iptables-alerter-script" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884246 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27614fa1-66da-4d21-a0d1-0ac85a9bdd13" volumeName="kubernetes.io/configmap/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-multus-daemon-config" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884259 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="526cc561-1bbd-4079-8a2f-326f939753e0" volumeName="kubernetes.io/secret/526cc561-1bbd-4079-8a2f-326f939753e0-image-registry-private-configuration" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884270 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="526cc561-1bbd-4079-8a2f-326f939753e0" volumeName="kubernetes.io/secret/526cc561-1bbd-4079-8a2f-326f939753e0-installation-pull-secrets" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884282 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fa688d8-29ca-4c1f-926a-f1a4099e6b44" volumeName="kubernetes.io/projected/9fa688d8-29ca-4c1f-926a-f1a4099e6b44-kube-api-access-6cv5p" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884293 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd8f9a4a-5b77-4a52-9ea9-a12245c00489" volumeName="kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-bound-sa-token" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884307 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fed8e88-bc49-4564-abcc-7d0fff068aa4" volumeName="kubernetes.io/empty-dir/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-tuned" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884320 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2277c280-7906-4c42-87f1-7941012a2d31" volumeName="kubernetes.io/configmap/2277c280-7906-4c42-87f1-7941012a2d31-ovnkube-script-lib" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884333 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d080b8e-cca6-4040-830e-cc9e9e25c50a" volumeName="kubernetes.io/projected/9d080b8e-cca6-4040-830e-cc9e9e25c50a-kube-api-access-6ht68" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884346 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd8f9a4a-5b77-4a52-9ea9-a12245c00489" volumeName="kubernetes.io/empty-dir/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-ca-trust-extracted" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884360 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18a4ebbe-c390-49d8-99e7-858bfaa3b834" volumeName="kubernetes.io/configmap/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-config" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884394 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2277c280-7906-4c42-87f1-7941012a2d31" volumeName="kubernetes.io/configmap/2277c280-7906-4c42-87f1-7941012a2d31-ovnkube-config" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884420 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30673bf9-d332-4592-afce-b16db49876cb" volumeName="kubernetes.io/projected/30673bf9-d332-4592-afce-b16db49876cb-kube-api-access-skw5p" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884434 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5166c4d7-2b82-4bbd-8d58-836cb274ed25" volumeName="kubernetes.io/secret/5166c4d7-2b82-4bbd-8d58-836cb274ed25-agent-certs" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884448 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27614fa1-66da-4d21-a0d1-0ac85a9bdd13" volumeName="kubernetes.io/projected/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-kube-api-access-wmw7p" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884494 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0440635e-2cf5-4f40-a4f5-e716376044ba" volumeName="kubernetes.io/projected/0440635e-2cf5-4f40-a4f5-e716376044ba-kube-api-access-7xgwg" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884509 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fed8e88-bc49-4564-abcc-7d0fff068aa4" volumeName="kubernetes.io/empty-dir/0fed8e88-bc49-4564-abcc-7d0fff068aa4-tmp" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884535 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2277c280-7906-4c42-87f1-7941012a2d31" volumeName="kubernetes.io/projected/2277c280-7906-4c42-87f1-7941012a2d31-kube-api-access-l48jx" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884550 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="526cc561-1bbd-4079-8a2f-326f939753e0" volumeName="kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-bound-sa-token" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884564 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a20efc24-e4cb-4075-911e-bfc43dcfc20b" volumeName="kubernetes.io/projected/a20efc24-e4cb-4075-911e-bfc43dcfc20b-kube-api-access-dpg2j" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884578 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de19ca9c-b22d-49a4-a5a6-a5f3a61bf892" volumeName="kubernetes.io/projected/de19ca9c-b22d-49a4-a5a6-a5f3a61bf892-kube-api-access-mglpv" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884622 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33f61ea8-5b1a-405a-aa36-7f07d79e2ec8" volumeName="kubernetes.io/configmap/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-service-ca-bundle" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884637 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33f61ea8-5b1a-405a-aa36-7f07d79e2ec8" volumeName="kubernetes.io/empty-dir/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-snapshots" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884650 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85b08631-ff08-461f-8bda-6a64a609e1f1" volumeName="kubernetes.io/configmap/85b08631-ff08-461f-8bda-6a64a609e1f1-serviceca" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884671 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a82abd0-5057-4361-9b0d-7a4f23ae97d1" volumeName="kubernetes.io/projected/8a82abd0-5057-4361-9b0d-7a4f23ae97d1-kube-api-access-xd5wb" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884684 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd8f9a4a-5b77-4a52-9ea9-a12245c00489" volumeName="kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-kube-api-access-f87wl" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884698 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09f5d7e2-51b8-4153-89c5-674d67065762" volumeName="kubernetes.io/empty-dir/09f5d7e2-51b8-4153-89c5-674d67065762-archives-path" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884713 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27614fa1-66da-4d21-a0d1-0ac85a9bdd13" volumeName="kubernetes.io/configmap/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-cni-binary-copy" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884727 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5166c4d7-2b82-4bbd-8d58-836cb274ed25" volumeName="kubernetes.io/configmap/5166c4d7-2b82-4bbd-8d58-836cb274ed25-konnectivity-ca" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884740 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3ce5e93-7d27-4fa6-b360-e571d8f18481" volumeName="kubernetes.io/projected/c3ce5e93-7d27-4fa6-b360-e571d8f18481-kube-api-access-v4snt" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884756 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4350294b-64f3-4efd-9619-90c013d82432" volumeName="kubernetes.io/projected/4350294b-64f3-4efd-9619-90c013d82432-kube-api-access-7jzxf" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884771 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85b08631-ff08-461f-8bda-6a64a609e1f1" volumeName="kubernetes.io/projected/85b08631-ff08-461f-8bda-6a64a609e1f1-kube-api-access-4g6m6" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884789 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d080b8e-cca6-4040-830e-cc9e9e25c50a" volumeName="kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-stats-auth" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884801 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0440635e-2cf5-4f40-a4f5-e716376044ba" volumeName="kubernetes.io/configmap/0440635e-2cf5-4f40-a4f5-e716376044ba-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884817 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2277c280-7906-4c42-87f1-7941012a2d31" volumeName="kubernetes.io/secret/2277c280-7906-4c42-87f1-7941012a2d31-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884833 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33f61ea8-5b1a-405a-aa36-7f07d79e2ec8" volumeName="kubernetes.io/configmap/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-trusted-ca-bundle" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884848 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="526cc561-1bbd-4079-8a2f-326f939753e0" volumeName="kubernetes.io/configmap/526cc561-1bbd-4079-8a2f-326f939753e0-registry-certificates" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884860 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d95c49a-0861-4878-b10f-6b51c391b19e" volumeName="kubernetes.io/projected/8d95c49a-0861-4878-b10f-6b51c391b19e-kube-api-access-dzmtr" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884875 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9e7a0c8-43b3-4f8c-8870-505edace7f78" volumeName="kubernetes.io/empty-dir/a9e7a0c8-43b3-4f8c-8870-505edace7f78-tmp-dir" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884888 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2413398-197b-4cad-8dc4-9bba9effe3b7" volumeName="kubernetes.io/projected/d2413398-197b-4cad-8dc4-9bba9effe3b7-kube-api-access-spcsf" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884948 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd8f9a4a-5b77-4a52-9ea9-a12245c00489" volumeName="kubernetes.io/configmap/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-certificates" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884963 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0440635e-2cf5-4f40-a4f5-e716376044ba" volumeName="kubernetes.io/configmap/0440635e-2cf5-4f40-a4f5-e716376044ba-cni-binary-copy" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884978 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32309642-7cba-4361-83b3-4680659c0544" volumeName="kubernetes.io/projected/32309642-7cba-4361-83b3-4680659c0544-kube-api-access-5dpbj" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.884991 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33f61ea8-5b1a-405a-aa36-7f07d79e2ec8" volumeName="kubernetes.io/projected/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-kube-api-access-6xrgq" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.885004 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a82abd0-5057-4361-9b0d-7a4f23ae97d1" volumeName="kubernetes.io/secret/8a82abd0-5057-4361-9b0d-7a4f23ae97d1-serving-cert" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.885017 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb8917db-c1ba-43af-995e-1948de69aa29" volumeName="kubernetes.io/secret/fb8917db-c1ba-43af-995e-1948de69aa29-original-pull-secret" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.885029 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd8f9a4a-5b77-4a52-9ea9-a12245c00489" volumeName="kubernetes.io/secret/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-image-registry-private-configuration" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.885041 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09f5d7e2-51b8-4153-89c5-674d67065762" volumeName="kubernetes.io/configmap/09f5d7e2-51b8-4153-89c5-674d67065762-service-ca-bundle" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.885054 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acc82f37-fad2-424d-89d0-5e5876cb75de" volumeName="kubernetes.io/empty-dir/acc82f37-fad2-424d-89d0-5e5876cb75de-tmp-dir" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.885068 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31c2770e-628c-4bcd-a088-73f190c952cd" volumeName="kubernetes.io/projected/31c2770e-628c-4bcd-a088-73f190c952cd-kube-api-access-s2dzh" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.885081 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="526cc561-1bbd-4079-8a2f-326f939753e0" volumeName="kubernetes.io/configmap/526cc561-1bbd-4079-8a2f-326f939753e0-trusted-ca" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.885095 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acc82f37-fad2-424d-89d0-5e5876cb75de" volumeName="kubernetes.io/configmap/acc82f37-fad2-424d-89d0-5e5876cb75de-config-volume" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.885108 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aff3a44b-6cad-449e-8733-f85e9899ecc3" volumeName="kubernetes.io/configmap/aff3a44b-6cad-449e-8733-f85e9899ecc3-telemetry-config" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.885120 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de19ca9c-b22d-49a4-a5a6-a5f3a61bf892" volumeName="kubernetes.io/secret/de19ca9c-b22d-49a4-a5a6-a5f3a61bf892-cert" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.885136 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd8f9a4a-5b77-4a52-9ea9-a12245c00489" volumeName="kubernetes.io/configmap/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-trusted-ca" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.885150 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09f5d7e2-51b8-4153-89c5-674d67065762" volumeName="kubernetes.io/projected/09f5d7e2-51b8-4153-89c5-674d67065762-kube-api-access-lgkdb" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.885164 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2277c280-7906-4c42-87f1-7941012a2d31" volumeName="kubernetes.io/configmap/2277c280-7906-4c42-87f1-7941012a2d31-env-overrides" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.885179 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33f61ea8-5b1a-405a-aa36-7f07d79e2ec8" volumeName="kubernetes.io/secret/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-serving-cert" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.885193 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d3be8a7-773b-42e1-a9a1-60d7df717301" volumeName="kubernetes.io/configmap/0d3be8a7-773b-42e1-a9a1-60d7df717301-nginx-conf" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.885206 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12aceb3a-2476-4efc-94d8-e6ec78cfc680" volumeName="kubernetes.io/empty-dir/12aceb3a-2476-4efc-94d8-e6ec78cfc680-data-volume" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.885220 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18a4ebbe-c390-49d8-99e7-858bfaa3b834" volumeName="kubernetes.io/projected/18a4ebbe-c390-49d8-99e7-858bfaa3b834-kube-api-access-s8xpk" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.885235 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33f61ea8-5b1a-405a-aa36-7f07d79e2ec8" volumeName="kubernetes.io/empty-dir/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-tmp" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.885248 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="526cc561-1bbd-4079-8a2f-326f939753e0" volumeName="kubernetes.io/empty-dir/526cc561-1bbd-4079-8a2f-326f939753e0-ca-trust-extracted" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.885262 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fa688d8-29ca-4c1f-926a-f1a4099e6b44" volumeName="kubernetes.io/secret/9fa688d8-29ca-4c1f-926a-f1a4099e6b44-serving-cert" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.885275 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10514d09-1dbf-47c2-bc1b-54a15f384cb3" volumeName="kubernetes.io/projected/10514d09-1dbf-47c2-bc1b-54a15f384cb3-kube-api-access-h6vsb" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.885288 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d080b8e-cca6-4040-830e-cc9e9e25c50a" volumeName="kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-default-certificate" seLinuxMountContext="" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.885301 5001 reconstruct.go:97] "Volume reconstruction finished" Oct 30 00:04:44.887486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.885308 5001 reconciler.go:29] "Reconciler: start to sync state" Oct 30 00:04:44.905437 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.905404 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8gb5_9fa688d8-29ca-4c1f-926a-f1a4099e6b44/console-operator/1.log" Oct 30 00:04:44.905584 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.905486 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8gb5_9fa688d8-29ca-4c1f-926a-f1a4099e6b44/console-operator/1.log" Oct 30 00:04:44.908162 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.908130 5001 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 00:04:44.909950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.909766 5001 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 00:04:44.909950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.909788 5001 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 00:04:44.909950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.909856 5001 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 00:04:44.910750 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:44.910720 5001 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 00:04:44.924107 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.924084 5001 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 00:04:44.926219 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.926195 5001 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:04:44.926366 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.926353 5001 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:04:44.929192 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.929145 5001 generic.go:356] "Generic (PLEG): container finished" podID="0440635e-2cf5-4f40-a4f5-e716376044ba" containerID="ef109cd5a18003bdc13d98e238f9a49667fee337c40c353ac5dd8ffd7b7937b0" exitCode=0 Oct 30 00:04:44.929192 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.929192 5001 generic.go:356] "Generic (PLEG): container finished" podID="0440635e-2cf5-4f40-a4f5-e716376044ba" containerID="f85a053acae262852b776bbc965ee2e7bf3055bc372018f74a4f39475502e1dd" exitCode=0 Oct 30 00:04:44.929355 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.929205 5001 generic.go:356] "Generic (PLEG): container finished" podID="0440635e-2cf5-4f40-a4f5-e716376044ba" containerID="97b4a76bd4f36ce5db8e919c83c8f46d42a090ea4dec95ad8df3158128f36946" exitCode=0 Oct 30 00:04:44.929355 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.929213 5001 generic.go:356] "Generic (PLEG): container finished" podID="0440635e-2cf5-4f40-a4f5-e716376044ba" containerID="1e56847d6524bcf16661d1b2c977c86d67a72f938ed7947dca9258561e5db08a" exitCode=0 Oct 30 00:04:44.929355 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.929221 5001 generic.go:356] "Generic (PLEG): container finished" podID="0440635e-2cf5-4f40-a4f5-e716376044ba" containerID="ce2133e430132ed8fc264df5d195ade3cd63bbcf9c3579f1aeb3480573c9f371" exitCode=0 Oct 30 00:04:44.929355 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.929229 5001 generic.go:356] "Generic (PLEG): container finished" podID="0440635e-2cf5-4f40-a4f5-e716376044ba" containerID="ac0536e875bde4894f14e79d5bff278d8724934dd74bd6288908f15c09fb34e0" exitCode=0 Oct 30 00:04:44.936318 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.936300 5001 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:04:44.936725 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.936707 5001 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:04:44.936808 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.936782 5001 generic.go:356] "Generic (PLEG): container finished" podID="eec55f048e12253ca5b78eb4bbe9a10b" containerID="7e94b648bc04fb39748d4f157f7d37462ff159ab123a148d6764252d93c7f4c4" exitCode=0 Oct 30 00:04:44.959380 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.959349 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8gb5_9fa688d8-29ca-4c1f-926a-f1a4099e6b44/console-operator/1.log" Oct 30 00:04:44.959540 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.959383 5001 generic.go:356] "Generic (PLEG): container finished" podID="9fa688d8-29ca-4c1f-926a-f1a4099e6b44" containerID="1b5598b379728a399ec94fb1bd83609608ffec6b569f58caae7897424bbb8d5f" exitCode=255 Oct 30 00:04:44.974782 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.974755 5001 manager.go:324] Recovery completed Oct 30 00:04:44.995490 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.995460 5001 cpu_manager.go:226] "Starting" policy="none" Oct 30 00:04:44.995490 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.995478 5001 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 00:04:44.995678 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.995500 5001 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 00:04:44.995678 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.995624 5001 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 00:04:44.996101 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.995635 5001 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 00:04:44.996140 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.996104 5001 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 00:04:44.996140 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.996111 5001 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 00:04:44.996140 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.996118 5001 policy_none.go:50] "Start" Oct 30 00:04:44.996140 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.996128 5001 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 00:04:44.996251 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.996141 5001 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 00:04:44.996291 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.996281 5001 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 00:04:44.996321 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.996292 5001 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 00:04:44.996321 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:44.996298 5001 policy_none.go:44] "Start" Oct 30 00:04:45.000667 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.000635 5001 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 00:04:45.003390 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.003376 5001 manager.go:341] "Starting Device Plugin manager" Oct 30 00:04:45.003454 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:45.003403 5001 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 00:04:45.003454 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.003412 5001 server.go:87] "Starting device plugin registration server" Oct 30 00:04:45.003753 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.003740 5001 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 00:04:45.003799 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.003754 5001 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 00:04:45.003852 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.003836 5001 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 00:04:45.003986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.003951 5001 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 00:04:45.004044 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.003997 5001 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 00:04:45.004044 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.004004 5001 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 00:04:45.004174 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.004074 5001 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T00:04:45.003981237Z","UUID":"5b04d118-b8f8-46ed-be64-a543aa1bd5d2","Handler":null,"Name":"","Endpoint":""} Oct 30 00:04:45.005516 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:45.005482 5001 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 00:04:45.005978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.005961 5001 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 00:04:45.006065 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.005988 5001 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 00:04:45.010843 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.010792 5001 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-120.ec2.internal"] Oct 30 00:04:45.011611 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.011592 5001 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal" Oct 30 00:04:45.011831 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.011791 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal" event={"ID":"eec55f048e12253ca5b78eb4bbe9a10b","Type":"ContainerStarted","Data":"73d4eb4fead93bc8647a781334f5b6ca16e62c7b95f4b1d47a32f764c41ae8ec"} Oct 30 00:04:45.011944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.011842 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal" event={"ID":"eec55f048e12253ca5b78eb4bbe9a10b","Type":"ContainerDied","Data":"7e94b648bc04fb39748d4f157f7d37462ff159ab123a148d6764252d93c7f4c4"} Oct 30 00:04:45.011944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.011864 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal" event={"ID":"eec55f048e12253ca5b78eb4bbe9a10b","Type":"ContainerStarted","Data":"b1ca01f6bedc6b25cc4de9362a8815a98eafbf74a13540072bca7da1aeefff30"} Oct 30 00:04:45.011944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.011895 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-120.ec2.internal" event={"ID":"4dc21da3207f1de3351a28354872ea48","Type":"ContainerStarted","Data":"72eb34cdcfe154e33becddd1ee0821fb2adfbbd8519818fdf2e2557a8d985f4c"} Oct 30 00:04:45.011944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.011936 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-120.ec2.internal" event={"ID":"4dc21da3207f1de3351a28354872ea48","Type":"ContainerStarted","Data":"2278a9b540030faf817efed78d6f518955498ae814e860c0c4a05446856e2b59"} Oct 30 00:04:45.012153 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.012136 5001 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-120.ec2.internal" Oct 30 00:04:45.032375 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.032353 5001 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:04:45.032468 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:45.032401 5001 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-120.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-120.ec2.internal" Oct 30 00:04:45.033114 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.033102 5001 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:04:45.033160 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:45.033129 5001 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal" Oct 30 00:04:45.104752 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.104674 5001 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:04:45.105963 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.105945 5001 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:04:45.106032 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.105978 5001 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:04:45.106032 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.105994 5001 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:04:45.106093 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.106051 5001 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-120.ec2.internal" Oct 30 00:04:45.124925 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.124885 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/eec55f048e12253ca5b78eb4bbe9a10b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal\" (UID: \"eec55f048e12253ca5b78eb4bbe9a10b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal" Oct 30 00:04:45.125020 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.124931 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eec55f048e12253ca5b78eb4bbe9a10b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal\" (UID: \"eec55f048e12253ca5b78eb4bbe9a10b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal" Oct 30 00:04:45.125020 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.124962 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4dc21da3207f1de3351a28354872ea48-config\") pod \"kube-apiserver-proxy-ip-10-0-143-120.ec2.internal\" (UID: \"4dc21da3207f1de3351a28354872ea48\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-120.ec2.internal" Oct 30 00:04:45.135750 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.135726 5001 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-120.ec2.internal" Oct 30 00:04:45.135846 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.135793 5001 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-120.ec2.internal" Oct 30 00:04:45.225520 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.225483 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/eec55f048e12253ca5b78eb4bbe9a10b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal\" (UID: \"eec55f048e12253ca5b78eb4bbe9a10b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal" Oct 30 00:04:45.225520 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.225517 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eec55f048e12253ca5b78eb4bbe9a10b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal\" (UID: \"eec55f048e12253ca5b78eb4bbe9a10b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal" Oct 30 00:04:45.225723 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.225537 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4dc21da3207f1de3351a28354872ea48-config\") pod \"kube-apiserver-proxy-ip-10-0-143-120.ec2.internal\" (UID: \"4dc21da3207f1de3351a28354872ea48\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-120.ec2.internal" Oct 30 00:04:45.225723 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.225590 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/eec55f048e12253ca5b78eb4bbe9a10b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal\" (UID: \"eec55f048e12253ca5b78eb4bbe9a10b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal" Oct 30 00:04:45.225723 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.225589 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4dc21da3207f1de3351a28354872ea48-config\") pod \"kube-apiserver-proxy-ip-10-0-143-120.ec2.internal\" (UID: \"4dc21da3207f1de3351a28354872ea48\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-120.ec2.internal" Oct 30 00:04:45.225723 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.225589 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eec55f048e12253ca5b78eb4bbe9a10b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal\" (UID: \"eec55f048e12253ca5b78eb4bbe9a10b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal" Oct 30 00:04:45.868948 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.868898 5001 apiserver.go:52] "Watching apiserver" Oct 30 00:04:45.912298 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.912271 5001 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 00:04:45.912903 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.912881 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mpg4t","openshift-insights/periodic-gathering-nkc49-p6m54","openshift-dns/node-resolver-b54v5","kube-system/kube-apiserver-proxy-ip-10-0-143-120.ec2.internal","openshift-image-registry/node-ca-qs54p","openshift-kube-storage-version-migrator/migrator-66df5d496b-xt8f2","openshift-network-console/networking-console-plugin-86c478bb88-7m2x2","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-n78wz","openshift-multus/network-metrics-daemon-52qpr","openshift-network-diagnostics/network-check-source-6f78d59f99-glf29","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l","openshift-network-diagnostics/network-check-target-sxvqx","kube-system/kas-connection-checker-685895c7c9-b9fcp","openshift-image-registry/image-registry-7c8c8757f-tspqd","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l25ts","kube-system/global-pull-secret-syncer-zm28x","openshift-monitoring/cluster-monitoring-operator-884576c57-nqw98","openshift-dns/dns-default-52kw4","kube-system/konnectivity-agent-q8m2x","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f","openshift-image-registry/image-registry-7dcddbb874-9zx5v","openshift-ovn-kubernetes/ovnkube-node-7k49v","openshift-multus/multus-56lxt","openshift-insights/insights-runtime-extractor-dd8wx","openshift-ingress/router-default-645c8fd54c-hp29d","openshift-insights/insights-operator-7c4986d449-nzxzl","openshift-multus/multus-additional-cni-plugins-l54xc","kube-system/kas-connection-checker-685895c7c9-4fpvt","kube-system/kas-connection-checker-685895c7c9-m98x7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal","openshift-network-operator/iptables-alerter-j6chh","openshift-cluster-node-tuning-operator/tuned-575b9","openshift-console-operator/console-operator-7cd56b8dbc-z8gb5","openshift-service-ca-operator/service-ca-operator-544d749c8c-6d24p"] Oct 30 00:04:45.916638 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.916610 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:45.917760 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.917738 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:45.918443 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.918424 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l25ts" Oct 30 00:04:45.919226 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.919210 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nqw98" Oct 30 00:04:45.919870 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.919855 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-7m2x2" Oct 30 00:04:45.924566 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.924549 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:45.925443 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.925429 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:45.925897 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.925884 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-52kw4" Oct 30 00:04:45.926231 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.926216 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 00:04:45.926279 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.926268 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 00:04:45.928222 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.928208 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:45.928570 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.928555 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 00:04:45.928625 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.928561 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:04:45.928790 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.928778 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 00:04:45.928870 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.928859 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 00:04:45.929010 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.928992 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 00:04:45.929382 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.929367 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 00:04:45.929421 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.929409 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 00:04:45.935193 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.935180 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 00:04:45.935519 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.935506 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 00:04:45.936681 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.936661 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 00:04:45.936681 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.936664 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 00:04:45.937335 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.937321 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 00:04:45.937591 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.937571 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 00:04:45.937715 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.937697 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-mjr9m\"" Oct 30 00:04:45.937764 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.937733 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 00:04:45.937882 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.937860 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wvd28\"" Oct 30 00:04:45.937882 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.937878 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-pxfs7\"" Oct 30 00:04:45.938152 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.938133 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 00:04:45.938673 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.938649 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 00:04:45.938801 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.938756 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 00:04:45.938926 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.938844 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 00:04:45.938926 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.938681 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:04:45.938926 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.938689 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 00:04:45.938926 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.938684 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 00:04:45.939129 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.939070 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 00:04:45.939359 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.939329 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 00:04:45.939696 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.939673 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 00:04:45.940247 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.940229 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kct9w\"" Oct 30 00:04:45.940330 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.940303 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 00:04:45.940330 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.940319 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 00:04:45.940593 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.940565 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 00:04:45.940703 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.940651 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 00:04:45.940823 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.940802 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 00:04:45.940990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.940967 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-lqg85\"" Oct 30 00:04:45.941058 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.940980 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 00:04:45.941058 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.941009 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-l5flb\"" Oct 30 00:04:45.941162 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.941114 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 00:04:45.941227 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.941209 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-cmxwh\"" Oct 30 00:04:45.941308 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.941291 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-n6hlg\"" Oct 30 00:04:45.941434 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.941409 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 00:04:45.941483 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.941438 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 00:04:45.941528 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.941497 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fmnxz\"" Oct 30 00:04:45.941759 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.941738 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 00:04:45.941819 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.941809 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 00:04:45.941868 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.941840 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-h92x8\"" Oct 30 00:04:45.942028 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.942013 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 00:04:45.942082 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.942052 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 00:04:45.942321 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.942301 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 00:04:45.942514 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.942480 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 00:04:45.942601 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.942567 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 00:04:45.942715 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.942699 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-j84nm\"" Oct 30 00:04:45.942851 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.942829 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 00:04:45.942949 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.942856 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 00:04:45.942949 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.942863 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 00:04:45.943036 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.942998 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-smkfn\"" Oct 30 00:04:45.943173 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.942698 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-k8n8m\"" Oct 30 00:04:45.943228 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.943201 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-r8lkn\"" Oct 30 00:04:45.943286 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.942482 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 00:04:45.943344 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.943330 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9vr82\"" Oct 30 00:04:45.943427 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.943409 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 00:04:45.943486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.943434 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bqlwc\"" Oct 30 00:04:45.943532 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.942838 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-n7fsr\"" Oct 30 00:04:45.943532 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.942831 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4xzk4\"" Oct 30 00:04:45.943532 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.943516 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 00:04:45.943667 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.943542 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ntcgl\"" Oct 30 00:04:45.943667 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.943545 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 00:04:45.943667 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.943628 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 00:04:45.943667 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.943629 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 00:04:45.943809 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.943698 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 00:04:45.943809 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.943716 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 00:04:45.943809 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.943739 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 00:04:45.943809 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.943741 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 00:04:45.943809 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.943792 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 00:04:45.944183 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.944162 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:04:45.944246 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.944167 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 00:04:45.944246 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.944232 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 00:04:45.944246 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.944232 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 00:04:45.944373 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.944257 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 00:04:45.944373 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.944276 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-m7wf2\"" Oct 30 00:04:45.944373 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.944303 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:04:45.944373 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.944304 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 00:04:45.944528 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.944452 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 00:04:45.944528 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.944507 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 00:04:45.944528 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.944525 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 00:04:45.944637 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.944591 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 00:04:45.944637 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.944615 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-thx59\"" Oct 30 00:04:45.946596 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.946569 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:04:45.946686 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.946607 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 00:04:45.946686 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.946607 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:04:45.946686 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.946584 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 00:04:45.946686 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.946623 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 00:04:45.946686 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.946638 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:04:45.946686 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.946639 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 00:04:45.947687 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.947674 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 00:04:45.947753 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.947706 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 00:04:45.947753 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.947708 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 00:04:45.947753 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.947708 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 00:04:45.948447 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.948434 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 00:04:45.956784 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.956761 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 00:04:45.956872 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.956860 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hslgr\"" Oct 30 00:04:45.957247 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.957234 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 00:04:45.957280 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.957253 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 00:04:45.965608 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.965592 5001 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-120.ec2.internal" Oct 30 00:04:45.965699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.965685 5001 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal" Oct 30 00:04:45.982119 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.982098 5001 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 00:04:45.984207 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.984192 5001 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:04:45.984262 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:45.984231 5001 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-120.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-120.ec2.internal" Oct 30 00:04:45.984598 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:45.984584 5001 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:04:45.984671 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:45.984624 5001 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal" Oct 30 00:04:46.028135 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.028108 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a9e7a0c8-43b3-4f8c-8870-505edace7f78-hosts-file\") pod \"node-resolver-b54v5\" (UID: \"a9e7a0c8-43b3-4f8c-8870-505edace7f78\") " pod="openshift-dns/node-resolver-b54v5" Oct 30 00:04:46.028249 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.028139 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-var-lib-kubelet\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.028249 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.028161 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-slash\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.028249 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.028183 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-var-lib-openvswitch\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.028249 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.028224 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-node-log\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.028456 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.028263 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-system-cni-dir\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.028456 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.028295 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0d3be8a7-773b-42e1-a9a1-60d7df717301-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7m2x2\" (UID: \"0d3be8a7-773b-42e1-a9a1-60d7df717301\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7m2x2" Oct 30 00:04:46.028456 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.028323 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-bound-sa-token\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:46.028456 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.028347 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6vsb\" (UniqueName: \"kubernetes.io/projected/10514d09-1dbf-47c2-bc1b-54a15f384cb3-kube-api-access-h6vsb\") pod \"iptables-alerter-j6chh\" (UID: \"10514d09-1dbf-47c2-bc1b-54a15f384cb3\") " pod="openshift-network-operator/iptables-alerter-j6chh" Oct 30 00:04:46.028456 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.028372 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-modprobe-d\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.028456 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.028400 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-kubelet\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.028456 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.028430 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2277c280-7906-4c42-87f1-7941012a2d31-env-overrides\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.028751 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.028487 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpg2j\" (UniqueName: \"kubernetes.io/projected/a20efc24-e4cb-4075-911e-bfc43dcfc20b-kube-api-access-dpg2j\") pod \"network-check-target-sxvqx\" (UID: \"a20efc24-e4cb-4075-911e-bfc43dcfc20b\") " pod="openshift-network-diagnostics/network-check-target-sxvqx" Oct 30 00:04:46.028751 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.028575 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cv5p\" (UniqueName: \"kubernetes.io/projected/9fa688d8-29ca-4c1f-926a-f1a4099e6b44-kube-api-access-6cv5p\") pod \"console-operator-7cd56b8dbc-z8gb5\" (UID: \"9fa688d8-29ca-4c1f-926a-f1a4099e6b44\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" Oct 30 00:04:46.028751 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.028577 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0d3be8a7-773b-42e1-a9a1-60d7df717301-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7m2x2\" (UID: \"0d3be8a7-773b-42e1-a9a1-60d7df717301\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7m2x2" Oct 30 00:04:46.028751 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.028619 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0440635e-2cf5-4f40-a4f5-e716376044ba-cni-binary-copy\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:46.028751 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.028650 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-sys\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.028751 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.028678 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09f5d7e2-51b8-4153-89c5-674d67065762-service-ca-bundle\") pod \"periodic-gathering-nkc49-p6m54\" (UID: \"09f5d7e2-51b8-4153-89c5-674d67065762\") " pod="openshift-insights/periodic-gathering-nkc49-p6m54" Oct 30 00:04:46.028751 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.028701 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09f5d7e2-51b8-4153-89c5-674d67065762-trusted-ca-bundle\") pod \"periodic-gathering-nkc49-p6m54\" (UID: \"09f5d7e2-51b8-4153-89c5-674d67065762\") " pod="openshift-insights/periodic-gathering-nkc49-p6m54" Oct 30 00:04:46.028751 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.028719 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-trusted-ca\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:46.028751 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.028741 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acc82f37-fad2-424d-89d0-5e5876cb75de-metrics-tls\") pod \"dns-default-52kw4\" (UID: \"acc82f37-fad2-424d-89d0-5e5876cb75de\") " pod="openshift-dns/dns-default-52kw4" Oct 30 00:04:46.029260 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.028770 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2277c280-7906-4c42-87f1-7941012a2d31-env-overrides\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.029260 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.028766 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5166c4d7-2b82-4bbd-8d58-836cb274ed25-konnectivity-ca\") pod \"konnectivity-agent-q8m2x\" (UID: \"5166c4d7-2b82-4bbd-8d58-836cb274ed25\") " pod="kube-system/konnectivity-agent-q8m2x" Oct 30 00:04:46.029260 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.028830 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09f5d7e2-51b8-4153-89c5-674d67065762-service-ca-bundle\") pod \"periodic-gathering-nkc49-p6m54\" (UID: \"09f5d7e2-51b8-4153-89c5-674d67065762\") " pod="openshift-insights/periodic-gathering-nkc49-p6m54" Oct 30 00:04:46.029260 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.028864 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-var-lib-cni-bin\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.029260 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.028893 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-multus-conf-dir\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.029260 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.028957 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a82abd0-5057-4361-9b0d-7a4f23ae97d1-config\") pod \"service-ca-operator-544d749c8c-6d24p\" (UID: \"8a82abd0-5057-4361-9b0d-7a4f23ae97d1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6d24p" Oct 30 00:04:46.029260 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029011 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0440635e-2cf5-4f40-a4f5-e716376044ba-cni-binary-copy\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:46.029260 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029015 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd5wb\" (UniqueName: \"kubernetes.io/projected/8a82abd0-5057-4361-9b0d-7a4f23ae97d1-kube-api-access-xd5wb\") pod \"service-ca-operator-544d749c8c-6d24p\" (UID: \"8a82abd0-5057-4361-9b0d-7a4f23ae97d1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6d24p" Oct 30 00:04:46.029260 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029056 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/526cc561-1bbd-4079-8a2f-326f939753e0-trusted-ca\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:46.029260 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029087 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-installation-pull-secrets\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:46.029260 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029119 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs\") pod \"network-metrics-daemon-52qpr\" (UID: \"30673bf9-d332-4592-afce-b16db49876cb\") " pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:46.029260 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029137 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a82abd0-5057-4361-9b0d-7a4f23ae97d1-config\") pod \"service-ca-operator-544d749c8c-6d24p\" (UID: \"8a82abd0-5057-4361-9b0d-7a4f23ae97d1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6d24p" Oct 30 00:04:46.029260 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029145 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0440635e-2cf5-4f40-a4f5-e716376044ba-cnibin\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:46.029260 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029163 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5166c4d7-2b82-4bbd-8d58-836cb274ed25-konnectivity-ca\") pod \"konnectivity-agent-q8m2x\" (UID: \"5166c4d7-2b82-4bbd-8d58-836cb274ed25\") " pod="kube-system/konnectivity-agent-q8m2x" Oct 30 00:04:46.029260 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029179 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0440635e-2cf5-4f40-a4f5-e716376044ba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:46.029260 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029246 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-metrics-certs\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:46.029934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029289 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-tuned\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.029934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029318 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fed8e88-bc49-4564-abcc-7d0fff068aa4-tmp\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.029934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029375 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-installation-pull-secrets\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:46.029934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029420 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-tuned\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.029934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029444 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2277c280-7906-4c42-87f1-7941012a2d31-ovn-node-metrics-cert\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.029934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029420 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fed8e88-bc49-4564-abcc-7d0fff068aa4-tmp\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.029934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029484 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0440635e-2cf5-4f40-a4f5-e716376044ba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:46.029934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029478 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/526cc561-1bbd-4079-8a2f-326f939753e0-image-registry-private-configuration\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:46.029934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029541 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/526cc561-1bbd-4079-8a2f-326f939753e0-ca-trust-extracted\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:46.029934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029571 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-tls\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:46.029934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029599 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-tmp\") pod \"insights-operator-7c4986d449-nzxzl\" (UID: \"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8\") " pod="openshift-insights/insights-operator-7c4986d449-nzxzl" Oct 30 00:04:46.029934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029627 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-sysctl-d\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.029934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029649 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2277c280-7906-4c42-87f1-7941012a2d31-ovn-node-metrics-cert\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.029934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029656 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/526cc561-1bbd-4079-8a2f-326f939753e0-image-registry-private-configuration\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:46.029934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029658 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/526cc561-1bbd-4079-8a2f-326f939753e0-ca-trust-extracted\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:46.029934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029652 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-bound-sa-token\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:46.029934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029749 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09f5d7e2-51b8-4153-89c5-674d67065762-trusted-ca-bundle\") pod \"periodic-gathering-nkc49-p6m54\" (UID: \"09f5d7e2-51b8-4153-89c5-674d67065762\") " pod="openshift-insights/periodic-gathering-nkc49-p6m54" Oct 30 00:04:46.029934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029776 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-trusted-ca\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:46.029934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029780 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfc4x\" (UniqueName: \"kubernetes.io/projected/acc82f37-fad2-424d-89d0-5e5876cb75de-kube-api-access-kfc4x\") pod \"dns-default-52kw4\" (UID: \"acc82f37-fad2-424d-89d0-5e5876cb75de\") " pod="openshift-dns/dns-default-52kw4" Oct 30 00:04:46.029934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029785 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-tmp\") pod \"insights-operator-7c4986d449-nzxzl\" (UID: \"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8\") " pod="openshift-insights/insights-operator-7c4986d449-nzxzl" Oct 30 00:04:46.029934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029811 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/526cc561-1bbd-4079-8a2f-326f939753e0-trusted-ca\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:46.029934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029835 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8xpk\" (UniqueName: \"kubernetes.io/projected/18a4ebbe-c390-49d8-99e7-858bfaa3b834-kube-api-access-s8xpk\") pod \"cluster-samples-operator-6ddb6c8b94-l25ts\" (UID: \"18a4ebbe-c390-49d8-99e7-858bfaa3b834\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l25ts" Oct 30 00:04:46.029934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029868 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xgwg\" (UniqueName: \"kubernetes.io/projected/0440635e-2cf5-4f40-a4f5-e716376044ba-kube-api-access-7xgwg\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:46.029934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029884 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-stats-auth\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:46.029934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029928 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs94l\" (UniqueName: \"kubernetes.io/projected/a9e7a0c8-43b3-4f8c-8870-505edace7f78-kube-api-access-cs94l\") pod \"node-resolver-b54v5\" (UID: \"a9e7a0c8-43b3-4f8c-8870-505edace7f78\") " pod="openshift-dns/node-resolver-b54v5" Oct 30 00:04:46.031014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029961 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-lib-modules\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.031014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.029990 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/85b08631-ff08-461f-8bda-6a64a609e1f1-serviceca\") pod \"node-ca-qs54p\" (UID: \"85b08631-ff08-461f-8bda-6a64a609e1f1\") " pod="openshift-image-registry/node-ca-qs54p" Oct 30 00:04:46.031014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.030016 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0440635e-2cf5-4f40-a4f5-e716376044ba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:46.031014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.030041 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10514d09-1dbf-47c2-bc1b-54a15f384cb3-host-slash\") pod \"iptables-alerter-j6chh\" (UID: \"10514d09-1dbf-47c2-bc1b-54a15f384cb3\") " pod="openshift-network-operator/iptables-alerter-j6chh" Oct 30 00:04:46.031014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.030086 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxvlt\" (UniqueName: \"kubernetes.io/projected/0fed8e88-bc49-4564-abcc-7d0fff068aa4-kube-api-access-qxvlt\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.031014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.030116 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-stats-auth\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:46.031014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.030119 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.031014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.030147 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-nzxzl\" (UID: \"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8\") " pod="openshift-insights/insights-operator-7c4986d449-nzxzl" Oct 30 00:04:46.031014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.030208 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-cni-binary-copy\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.031014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.030213 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/85b08631-ff08-461f-8bda-6a64a609e1f1-serviceca\") pod \"node-ca-qs54p\" (UID: \"85b08631-ff08-461f-8bda-6a64a609e1f1\") " pod="openshift-image-registry/node-ca-qs54p" Oct 30 00:04:46.031014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.030235 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-etc-kubernetes\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.031014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.030260 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:46.031014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.030285 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-socket-dir\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:46.031014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.030314 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgkdb\" (UniqueName: \"kubernetes.io/projected/09f5d7e2-51b8-4153-89c5-674d67065762-kube-api-access-lgkdb\") pod \"periodic-gathering-nkc49-p6m54\" (UID: \"09f5d7e2-51b8-4153-89c5-674d67065762\") " pod="openshift-insights/periodic-gathering-nkc49-p6m54" Oct 30 00:04:46.031014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.030326 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0440635e-2cf5-4f40-a4f5-e716376044ba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:46.031014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.030352 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/12aceb3a-2476-4efc-94d8-e6ec78cfc680-data-volume\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:46.031014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.030498 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-cni-binary-copy\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.031014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.030515 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-nzxzl\" (UID: \"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8\") " pod="openshift-insights/insights-operator-7c4986d449-nzxzl" Oct 30 00:04:46.031014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.030519 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spcsf\" (UniqueName: \"kubernetes.io/projected/d2413398-197b-4cad-8dc4-9bba9effe3b7-kube-api-access-spcsf\") pod \"kas-connection-checker-685895c7c9-m98x7\" (UID: \"d2413398-197b-4cad-8dc4-9bba9effe3b7\") " pod="kube-system/kas-connection-checker-685895c7c9-m98x7" Oct 30 00:04:46.031014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.030575 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/12aceb3a-2476-4efc-94d8-e6ec78cfc680-data-volume\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:46.031014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.030596 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb8917db-c1ba-43af-995e-1948de69aa29-dbus\") pod \"global-pull-secret-syncer-zm28x\" (UID: \"fb8917db-c1ba-43af-995e-1948de69aa29\") " pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:46.031014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.030631 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5166c4d7-2b82-4bbd-8d58-836cb274ed25-agent-certs\") pod \"konnectivity-agent-q8m2x\" (UID: \"5166c4d7-2b82-4bbd-8d58-836cb274ed25\") " pod="kube-system/konnectivity-agent-q8m2x" Oct 30 00:04:46.031014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.030702 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/12aceb3a-2476-4efc-94d8-e6ec78cfc680-crio-socket\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:46.031014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.030749 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-sysconfig\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.031014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.030774 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-kubernetes\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.031014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.030804 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-image-registry-private-configuration\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:46.031014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.030838 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5166c4d7-2b82-4bbd-8d58-836cb274ed25-agent-certs\") pod \"konnectivity-agent-q8m2x\" (UID: \"5166c4d7-2b82-4bbd-8d58-836cb274ed25\") " pod="kube-system/konnectivity-agent-q8m2x" Oct 30 00:04:46.031014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.030849 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v899\" (UniqueName: \"kubernetes.io/projected/c1a647f1-e715-4a8a-8b44-0196b3b1af27-kube-api-access-2v899\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l\" (UID: \"c1a647f1-e715-4a8a-8b44-0196b3b1af27\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l" Oct 30 00:04:46.031014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.030946 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-l25ts\" (UID: \"18a4ebbe-c390-49d8-99e7-858bfaa3b834\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l25ts" Oct 30 00:04:46.031014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.030981 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-image-registry-private-configuration\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.031053 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-run-netns\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.031071 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skw5p\" (UniqueName: \"kubernetes.io/projected/30673bf9-d332-4592-afce-b16db49876cb-kube-api-access-skw5p\") pod \"network-metrics-daemon-52qpr\" (UID: \"30673bf9-d332-4592-afce-b16db49876cb\") " pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.031102 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-cni-netd\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.031128 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de19ca9c-b22d-49a4-a5a6-a5f3a61bf892-cert\") pod \"ingress-canary-mpg4t\" (UID: \"de19ca9c-b22d-49a4-a5a6-a5f3a61bf892\") " pod="openshift-ingress-canary/ingress-canary-mpg4t" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.031231 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-l25ts\" (UID: \"18a4ebbe-c390-49d8-99e7-858bfaa3b834\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l25ts" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.031264 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-sys-fs\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.031289 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-service-ca-bundle\") pod \"insights-operator-7c4986d449-nzxzl\" (UID: \"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8\") " pod="openshift-insights/insights-operator-7c4986d449-nzxzl" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.031326 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g6m6\" (UniqueName: \"kubernetes.io/projected/85b08631-ff08-461f-8bda-6a64a609e1f1-kube-api-access-4g6m6\") pod \"node-ca-qs54p\" (UID: \"85b08631-ff08-461f-8bda-6a64a609e1f1\") " pod="openshift-image-registry/node-ca-qs54p" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.031357 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de19ca9c-b22d-49a4-a5a6-a5f3a61bf892-cert\") pod \"ingress-canary-mpg4t\" (UID: \"de19ca9c-b22d-49a4-a5a6-a5f3a61bf892\") " pod="openshift-ingress-canary/ingress-canary-mpg4t" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.031389 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-cnibin\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.031413 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-multus-daemon-config\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.031429 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-default-certificate\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.031441 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-service-ca-bundle\") pod \"insights-operator-7c4986d449-nzxzl\" (UID: \"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8\") " pod="openshift-insights/insights-operator-7c4986d449-nzxzl" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.031447 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1a647f1-e715-4a8a-8b44-0196b3b1af27-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l\" (UID: \"c1a647f1-e715-4a8a-8b44-0196b3b1af27\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.031468 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddgh4\" (UniqueName: \"kubernetes.io/projected/6f2b3396-a95d-4bc7-902f-1daf3dd689fd-kube-api-access-ddgh4\") pod \"kas-connection-checker-685895c7c9-b9fcp\" (UID: \"6f2b3396-a95d-4bc7-902f-1daf3dd689fd\") " pod="kube-system/kas-connection-checker-685895c7c9-b9fcp" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.031490 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-multus-socket-dir-parent\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.031536 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-host\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.031578 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-multus-daemon-config\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.031593 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-default-certificate\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.031633 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/aff3a44b-6cad-449e-8733-f85e9899ecc3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-nqw98\" (UID: \"aff3a44b-6cad-449e-8733-f85e9899ecc3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nqw98" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.031659 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1a647f1-e715-4a8a-8b44-0196b3b1af27-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l\" (UID: \"c1a647f1-e715-4a8a-8b44-0196b3b1af27\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.031663 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/09f5d7e2-51b8-4153-89c5-674d67065762-archives-path\") pod \"periodic-gathering-nkc49-p6m54\" (UID: \"09f5d7e2-51b8-4153-89c5-674d67065762\") " pod="openshift-insights/periodic-gathering-nkc49-p6m54" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.031688 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzmtr\" (UniqueName: \"kubernetes.io/projected/8d95c49a-0861-4878-b10f-6b51c391b19e-kube-api-access-dzmtr\") pod \"migrator-66df5d496b-xt8f2\" (UID: \"8d95c49a-0861-4878-b10f-6b51c391b19e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xt8f2" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.031715 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-systemd\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.031790 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/09f5d7e2-51b8-4153-89c5-674d67065762-archives-path\") pod \"periodic-gathering-nkc49-p6m54\" (UID: \"09f5d7e2-51b8-4153-89c5-674d67065762\") " pod="openshift-insights/periodic-gathering-nkc49-p6m54" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.031828 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb8917db-c1ba-43af-995e-1948de69aa29-original-pull-secret\") pod \"global-pull-secret-syncer-zm28x\" (UID: \"fb8917db-c1ba-43af-995e-1948de69aa29\") " pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.031860 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9fa688d8-29ca-4c1f-926a-f1a4099e6b44-config\") pod \"console-operator-7cd56b8dbc-z8gb5\" (UID: \"9fa688d8-29ca-4c1f-926a-f1a4099e6b44\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.031878 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/aff3a44b-6cad-449e-8733-f85e9899ecc3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-nqw98\" (UID: \"aff3a44b-6cad-449e-8733-f85e9899ecc3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nqw98" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.031888 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-serving-cert\") pod \"insights-operator-7c4986d449-nzxzl\" (UID: \"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8\") " pod="openshift-insights/insights-operator-7c4986d449-nzxzl" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.031926 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9e7a0c8-43b3-4f8c-8870-505edace7f78-tmp-dir\") pod \"node-resolver-b54v5\" (UID: \"a9e7a0c8-43b3-4f8c-8870-505edace7f78\") " pod="openshift-dns/node-resolver-b54v5" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.032006 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9e7a0c8-43b3-4f8c-8870-505edace7f78-tmp-dir\") pod \"node-resolver-b54v5\" (UID: \"a9e7a0c8-43b3-4f8c-8870-505edace7f78\") " pod="openshift-dns/node-resolver-b54v5" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.032017 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-sysctl-conf\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.032028 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9fa688d8-29ca-4c1f-926a-f1a4099e6b44-config\") pod \"console-operator-7cd56b8dbc-z8gb5\" (UID: \"9fa688d8-29ca-4c1f-926a-f1a4099e6b44\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.032037 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-run-netns\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.032040 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-serving-cert\") pod \"insights-operator-7c4986d449-nzxzl\" (UID: \"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8\") " pod="openshift-insights/insights-operator-7c4986d449-nzxzl" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.032079 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-run-openvswitch\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.032096 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l25ts\" (UID: \"18a4ebbe-c390-49d8-99e7-858bfaa3b834\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l25ts" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.032137 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb8917db-c1ba-43af-995e-1948de69aa29-original-pull-secret\") pod \"global-pull-secret-syncer-zm28x\" (UID: \"fb8917db-c1ba-43af-995e-1948de69aa29\") " pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.032138 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9fa688d8-29ca-4c1f-926a-f1a4099e6b44-trusted-ca\") pod \"console-operator-7cd56b8dbc-z8gb5\" (UID: \"9fa688d8-29ca-4c1f-926a-f1a4099e6b44\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.032174 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2dzh\" (UniqueName: \"kubernetes.io/projected/31c2770e-628c-4bcd-a088-73f190c952cd-kube-api-access-s2dzh\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.032225 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0440635e-2cf5-4f40-a4f5-e716376044ba-system-cni-dir\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.032310 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-run\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.032295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.032328 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2277c280-7906-4c42-87f1-7941012a2d31-ovnkube-script-lib\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.034080 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.032774 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0440635e-2cf5-4f40-a4f5-e716376044ba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:46.034080 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.032795 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9fa688d8-29ca-4c1f-926a-f1a4099e6b44-trusted-ca\") pod \"console-operator-7cd56b8dbc-z8gb5\" (UID: \"9fa688d8-29ca-4c1f-926a-f1a4099e6b44\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" Oct 30 00:04:46.034080 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.032819 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9fa688d8-29ca-4c1f-926a-f1a4099e6b44-serving-cert\") pod \"console-operator-7cd56b8dbc-z8gb5\" (UID: \"9fa688d8-29ca-4c1f-926a-f1a4099e6b44\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" Oct 30 00:04:46.034080 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.032857 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-run-k8s-cni-cncf-io\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.034080 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.032860 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2277c280-7906-4c42-87f1-7941012a2d31-ovnkube-script-lib\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.036154 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.036064 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d080b8e-cca6-4040-830e-cc9e9e25c50a-service-ca-bundle\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:46.036473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.036454 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/10514d09-1dbf-47c2-bc1b-54a15f384cb3-iptables-alerter-script\") pod \"iptables-alerter-j6chh\" (UID: \"10514d09-1dbf-47c2-bc1b-54a15f384cb3\") " pod="openshift-network-operator/iptables-alerter-j6chh" Oct 30 00:04:46.036546 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.036488 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/acc82f37-fad2-424d-89d0-5e5876cb75de-config-volume\") pod \"dns-default-52kw4\" (UID: \"acc82f37-fad2-424d-89d0-5e5876cb75de\") " pod="openshift-dns/dns-default-52kw4" Oct 30 00:04:46.036546 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.036506 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xrgq\" (UniqueName: \"kubernetes.io/projected/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-kube-api-access-6xrgq\") pod \"insights-operator-7c4986d449-nzxzl\" (UID: \"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8\") " pod="openshift-insights/insights-operator-7c4986d449-nzxzl" Oct 30 00:04:46.036546 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.036508 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9fa688d8-29ca-4c1f-926a-f1a4099e6b44-serving-cert\") pod \"console-operator-7cd56b8dbc-z8gb5\" (UID: \"9fa688d8-29ca-4c1f-926a-f1a4099e6b44\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" Oct 30 00:04:46.036546 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.036523 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-cni-bin\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.036706 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.036553 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-multus-cni-dir\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.036706 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.036611 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-var-lib-kubelet\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.036706 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.036647 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-etc-openvswitch\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.036706 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.036664 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/10514d09-1dbf-47c2-bc1b-54a15f384cb3-iptables-alerter-script\") pod \"iptables-alerter-j6chh\" (UID: \"10514d09-1dbf-47c2-bc1b-54a15f384cb3\") " pod="openshift-network-operator/iptables-alerter-j6chh" Oct 30 00:04:46.036706 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.036682 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/acc82f37-fad2-424d-89d0-5e5876cb75de-config-volume\") pod \"dns-default-52kw4\" (UID: \"acc82f37-fad2-424d-89d0-5e5876cb75de\") " pod="openshift-dns/dns-default-52kw4" Oct 30 00:04:46.036931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.036719 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a82abd0-5057-4361-9b0d-7a4f23ae97d1-serving-cert\") pod \"service-ca-operator-544d749c8c-6d24p\" (UID: \"8a82abd0-5057-4361-9b0d-7a4f23ae97d1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6d24p" Oct 30 00:04:46.036931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.036754 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cngbs\" (UniqueName: \"kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-kube-api-access-cngbs\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:46.036931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.036771 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-certificates\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:46.036931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.036792 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-log-socket\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.036931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.036798 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a82abd0-5057-4361-9b0d-7a4f23ae97d1-serving-cert\") pod \"service-ca-operator-544d749c8c-6d24p\" (UID: \"8a82abd0-5057-4361-9b0d-7a4f23ae97d1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6d24p" Oct 30 00:04:46.036931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.036815 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f87wl\" (UniqueName: \"kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-kube-api-access-f87wl\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:46.037187 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.036957 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1a647f1-e715-4a8a-8b44-0196b3b1af27-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l\" (UID: \"c1a647f1-e715-4a8a-8b44-0196b3b1af27\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l" Oct 30 00:04:46.037187 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.037028 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-run-systemd\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.037187 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.037032 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-certificates\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:46.037187 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.037060 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-run-ovn-kubernetes\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.037187 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.037101 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l48jx\" (UniqueName: \"kubernetes.io/projected/2277c280-7906-4c42-87f1-7941012a2d31-kube-api-access-l48jx\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.037187 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.037107 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1a647f1-e715-4a8a-8b44-0196b3b1af27-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l\" (UID: \"c1a647f1-e715-4a8a-8b44-0196b3b1af27\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l" Oct 30 00:04:46.037187 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.037129 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-ca-trust-extracted\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:46.037454 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.037202 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-ca-trust-extracted\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:46.037454 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.037247 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2277c280-7906-4c42-87f1-7941012a2d31-ovnkube-config\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.037454 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.037269 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4snt\" (UniqueName: \"kubernetes.io/projected/c3ce5e93-7d27-4fa6-b360-e571d8f18481-kube-api-access-v4snt\") pod \"kas-connection-checker-685895c7c9-4fpvt\" (UID: \"c3ce5e93-7d27-4fa6-b360-e571d8f18481\") " pod="kube-system/kas-connection-checker-685895c7c9-4fpvt" Oct 30 00:04:46.037454 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.037285 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-var-lib-cni-multus\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.037454 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.037302 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmw7p\" (UniqueName: \"kubernetes.io/projected/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-kube-api-access-wmw7p\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.037676 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.037472 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ht68\" (UniqueName: \"kubernetes.io/projected/9d080b8e-cca6-4040-830e-cc9e9e25c50a-kube-api-access-6ht68\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:46.037676 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.037500 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/526cc561-1bbd-4079-8a2f-326f939753e0-registry-certificates\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:46.037676 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.037477 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2277c280-7906-4c42-87f1-7941012a2d31-ovnkube-config\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.037676 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.037517 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-snapshots\") pod \"insights-operator-7c4986d449-nzxzl\" (UID: \"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8\") " pod="openshift-insights/insights-operator-7c4986d449-nzxzl" Oct 30 00:04:46.037676 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.037568 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-snapshots\") pod \"insights-operator-7c4986d449-nzxzl\" (UID: \"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8\") " pod="openshift-insights/insights-operator-7c4986d449-nzxzl" Oct 30 00:04:46.037676 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.037566 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-run-multus-certs\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.037676 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.037612 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12aceb3a-2476-4efc-94d8-e6ec78cfc680-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:46.037676 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.037623 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/526cc561-1bbd-4079-8a2f-326f939753e0-registry-certificates\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:46.037676 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.037647 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-run-ovn\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.037676 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.037675 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acc82f37-fad2-424d-89d0-5e5876cb75de-tmp-dir\") pod \"dns-default-52kw4\" (UID: \"acc82f37-fad2-424d-89d0-5e5876cb75de\") " pod="openshift-dns/dns-default-52kw4" Oct 30 00:04:46.038055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.037692 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7m2x2\" (UID: \"0d3be8a7-773b-42e1-a9a1-60d7df717301\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7m2x2" Oct 30 00:04:46.038055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.037708 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12aceb3a-2476-4efc-94d8-e6ec78cfc680-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:46.038055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.037725 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jzxf\" (UniqueName: \"kubernetes.io/projected/4350294b-64f3-4efd-9619-90c013d82432-kube-api-access-7jzxf\") pod \"volume-data-source-validator-58c9f5f46d-n78wz\" (UID: \"4350294b-64f3-4efd-9619-90c013d82432\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-n78wz" Oct 30 00:04:46.038055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.037733 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acc82f37-fad2-424d-89d0-5e5876cb75de-tmp-dir\") pod \"dns-default-52kw4\" (UID: \"acc82f37-fad2-424d-89d0-5e5876cb75de\") " pod="openshift-dns/dns-default-52kw4" Oct 30 00:04:46.038055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.037740 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-etc-selinux\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:46.038055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.037755 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mglpv\" (UniqueName: \"kubernetes.io/projected/de19ca9c-b22d-49a4-a5a6-a5f3a61bf892-kube-api-access-mglpv\") pod \"ingress-canary-mpg4t\" (UID: \"de19ca9c-b22d-49a4-a5a6-a5f3a61bf892\") " pod="openshift-ingress-canary/ingress-canary-mpg4t" Oct 30 00:04:46.038055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.037779 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/526cc561-1bbd-4079-8a2f-326f939753e0-installation-pull-secrets\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:46.038055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.037884 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-systemd-units\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.038055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.037937 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-registry-tls\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:46.038055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.037972 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-hostroot\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.038055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.037987 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/526cc561-1bbd-4079-8a2f-326f939753e0-installation-pull-secrets\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:46.038055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.038019 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vrjg\" (UniqueName: \"kubernetes.io/projected/12aceb3a-2476-4efc-94d8-e6ec78cfc680-kube-api-access-9vrjg\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:46.038055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.038047 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb8917db-c1ba-43af-995e-1948de69aa29-kubelet-config\") pod \"global-pull-secret-syncer-zm28x\" (UID: \"fb8917db-c1ba-43af-995e-1948de69aa29\") " pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:46.038427 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.038068 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aff3a44b-6cad-449e-8733-f85e9899ecc3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nqw98\" (UID: \"aff3a44b-6cad-449e-8733-f85e9899ecc3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nqw98" Oct 30 00:04:46.038427 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.038169 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56j8l\" (UniqueName: \"kubernetes.io/projected/aff3a44b-6cad-449e-8733-f85e9899ecc3-kube-api-access-56j8l\") pod \"cluster-monitoring-operator-884576c57-nqw98\" (UID: \"aff3a44b-6cad-449e-8733-f85e9899ecc3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nqw98" Oct 30 00:04:46.038427 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.038200 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-device-dir\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:46.038427 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.038221 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-registration-dir\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:46.038427 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.038238 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dpbj\" (UniqueName: \"kubernetes.io/projected/32309642-7cba-4361-83b3-4680659c0544-kube-api-access-5dpbj\") pod \"network-check-source-6f78d59f99-glf29\" (UID: \"32309642-7cba-4361-83b3-4680659c0544\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-glf29" Oct 30 00:04:46.038427 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.038254 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85b08631-ff08-461f-8bda-6a64a609e1f1-host\") pod \"node-ca-qs54p\" (UID: \"85b08631-ff08-461f-8bda-6a64a609e1f1\") " pod="openshift-image-registry/node-ca-qs54p" Oct 30 00:04:46.047597 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.047579 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd5wb\" (UniqueName: \"kubernetes.io/projected/8a82abd0-5057-4361-9b0d-7a4f23ae97d1-kube-api-access-xd5wb\") pod \"service-ca-operator-544d749c8c-6d24p\" (UID: \"8a82abd0-5057-4361-9b0d-7a4f23ae97d1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6d24p" Oct 30 00:04:46.048320 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.048294 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-bound-sa-token\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:46.048408 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.048331 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6vsb\" (UniqueName: \"kubernetes.io/projected/10514d09-1dbf-47c2-bc1b-54a15f384cb3-kube-api-access-h6vsb\") pod \"iptables-alerter-j6chh\" (UID: \"10514d09-1dbf-47c2-bc1b-54a15f384cb3\") " pod="openshift-network-operator/iptables-alerter-j6chh" Oct 30 00:04:46.048502 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.048483 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpg2j\" (UniqueName: \"kubernetes.io/projected/a20efc24-e4cb-4075-911e-bfc43dcfc20b-kube-api-access-dpg2j\") pod \"network-check-target-sxvqx\" (UID: \"a20efc24-e4cb-4075-911e-bfc43dcfc20b\") " pod="openshift-network-diagnostics/network-check-target-sxvqx" Oct 30 00:04:46.048556 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.048533 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cv5p\" (UniqueName: \"kubernetes.io/projected/9fa688d8-29ca-4c1f-926a-f1a4099e6b44-kube-api-access-6cv5p\") pod \"console-operator-7cd56b8dbc-z8gb5\" (UID: \"9fa688d8-29ca-4c1f-926a-f1a4099e6b44\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" Oct 30 00:04:46.050396 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.050381 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-bound-sa-token\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:46.050595 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.050577 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xgwg\" (UniqueName: \"kubernetes.io/projected/0440635e-2cf5-4f40-a4f5-e716376044ba-kube-api-access-7xgwg\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:46.051190 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.051173 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfc4x\" (UniqueName: \"kubernetes.io/projected/acc82f37-fad2-424d-89d0-5e5876cb75de-kube-api-access-kfc4x\") pod \"dns-default-52kw4\" (UID: \"acc82f37-fad2-424d-89d0-5e5876cb75de\") " pod="openshift-dns/dns-default-52kw4" Oct 30 00:04:46.051190 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.051182 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8xpk\" (UniqueName: \"kubernetes.io/projected/18a4ebbe-c390-49d8-99e7-858bfaa3b834-kube-api-access-s8xpk\") pod \"cluster-samples-operator-6ddb6c8b94-l25ts\" (UID: \"18a4ebbe-c390-49d8-99e7-858bfaa3b834\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l25ts" Oct 30 00:04:46.053843 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.053827 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g6m6\" (UniqueName: \"kubernetes.io/projected/85b08631-ff08-461f-8bda-6a64a609e1f1-kube-api-access-4g6m6\") pod \"node-ca-qs54p\" (UID: \"85b08631-ff08-461f-8bda-6a64a609e1f1\") " pod="openshift-image-registry/node-ca-qs54p" Oct 30 00:04:46.054132 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.054105 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddgh4\" (UniqueName: \"kubernetes.io/projected/6f2b3396-a95d-4bc7-902f-1daf3dd689fd-kube-api-access-ddgh4\") pod \"kas-connection-checker-685895c7c9-b9fcp\" (UID: \"6f2b3396-a95d-4bc7-902f-1daf3dd689fd\") " pod="kube-system/kas-connection-checker-685895c7c9-b9fcp" Oct 30 00:04:46.054343 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.054326 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2dzh\" (UniqueName: \"kubernetes.io/projected/31c2770e-628c-4bcd-a088-73f190c952cd-kube-api-access-s2dzh\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:46.054504 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.054478 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxvlt\" (UniqueName: \"kubernetes.io/projected/0fed8e88-bc49-4564-abcc-7d0fff068aa4-kube-api-access-qxvlt\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.054641 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.054624 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzmtr\" (UniqueName: \"kubernetes.io/projected/8d95c49a-0861-4878-b10f-6b51c391b19e-kube-api-access-dzmtr\") pod \"migrator-66df5d496b-xt8f2\" (UID: \"8d95c49a-0861-4878-b10f-6b51c391b19e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xt8f2" Oct 30 00:04:46.057053 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.057037 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v899\" (UniqueName: \"kubernetes.io/projected/c1a647f1-e715-4a8a-8b44-0196b3b1af27-kube-api-access-2v899\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l\" (UID: \"c1a647f1-e715-4a8a-8b44-0196b3b1af27\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l" Oct 30 00:04:46.059245 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.059227 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgkdb\" (UniqueName: \"kubernetes.io/projected/09f5d7e2-51b8-4153-89c5-674d67065762-kube-api-access-lgkdb\") pod \"periodic-gathering-nkc49-p6m54\" (UID: \"09f5d7e2-51b8-4153-89c5-674d67065762\") " pod="openshift-insights/periodic-gathering-nkc49-p6m54" Oct 30 00:04:46.059306 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.059265 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spcsf\" (UniqueName: \"kubernetes.io/projected/d2413398-197b-4cad-8dc4-9bba9effe3b7-kube-api-access-spcsf\") pod \"kas-connection-checker-685895c7c9-m98x7\" (UID: \"d2413398-197b-4cad-8dc4-9bba9effe3b7\") " pod="kube-system/kas-connection-checker-685895c7c9-m98x7" Oct 30 00:04:46.059369 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.059352 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4snt\" (UniqueName: \"kubernetes.io/projected/c3ce5e93-7d27-4fa6-b360-e571d8f18481-kube-api-access-v4snt\") pod \"kas-connection-checker-685895c7c9-4fpvt\" (UID: \"c3ce5e93-7d27-4fa6-b360-e571d8f18481\") " pod="kube-system/kas-connection-checker-685895c7c9-4fpvt" Oct 30 00:04:46.059634 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.059614 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cngbs\" (UniqueName: \"kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-kube-api-access-cngbs\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:46.059733 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.059718 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmw7p\" (UniqueName: \"kubernetes.io/projected/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-kube-api-access-wmw7p\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.059792 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.059777 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skw5p\" (UniqueName: \"kubernetes.io/projected/30673bf9-d332-4592-afce-b16db49876cb-kube-api-access-skw5p\") pod \"network-metrics-daemon-52qpr\" (UID: \"30673bf9-d332-4592-afce-b16db49876cb\") " pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:46.061448 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.061419 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f87wl\" (UniqueName: \"kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-kube-api-access-f87wl\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:46.062541 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.062525 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ht68\" (UniqueName: \"kubernetes.io/projected/9d080b8e-cca6-4040-830e-cc9e9e25c50a-kube-api-access-6ht68\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:46.062657 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.062642 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xrgq\" (UniqueName: \"kubernetes.io/projected/33f61ea8-5b1a-405a-aa36-7f07d79e2ec8-kube-api-access-6xrgq\") pod \"insights-operator-7c4986d449-nzxzl\" (UID: \"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8\") " pod="openshift-insights/insights-operator-7c4986d449-nzxzl" Oct 30 00:04:46.064126 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.064110 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jzxf\" (UniqueName: \"kubernetes.io/projected/4350294b-64f3-4efd-9619-90c013d82432-kube-api-access-7jzxf\") pod \"volume-data-source-validator-58c9f5f46d-n78wz\" (UID: \"4350294b-64f3-4efd-9619-90c013d82432\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-n78wz" Oct 30 00:04:46.064353 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.064331 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l48jx\" (UniqueName: \"kubernetes.io/projected/2277c280-7906-4c42-87f1-7941012a2d31-kube-api-access-l48jx\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.064712 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.064694 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vrjg\" (UniqueName: \"kubernetes.io/projected/12aceb3a-2476-4efc-94d8-e6ec78cfc680-kube-api-access-9vrjg\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:46.064776 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.064713 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dpbj\" (UniqueName: \"kubernetes.io/projected/32309642-7cba-4361-83b3-4680659c0544-kube-api-access-5dpbj\") pod \"network-check-source-6f78d59f99-glf29\" (UID: \"32309642-7cba-4361-83b3-4680659c0544\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-glf29" Oct 30 00:04:46.064895 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.064879 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs94l\" (UniqueName: \"kubernetes.io/projected/a9e7a0c8-43b3-4f8c-8870-505edace7f78-kube-api-access-cs94l\") pod \"node-resolver-b54v5\" (UID: \"a9e7a0c8-43b3-4f8c-8870-505edace7f78\") " pod="openshift-dns/node-resolver-b54v5" Oct 30 00:04:46.064952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.064905 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mglpv\" (UniqueName: \"kubernetes.io/projected/de19ca9c-b22d-49a4-a5a6-a5f3a61bf892-kube-api-access-mglpv\") pod \"ingress-canary-mpg4t\" (UID: \"de19ca9c-b22d-49a4-a5a6-a5f3a61bf892\") " pod="openshift-ingress-canary/ingress-canary-mpg4t" Oct 30 00:04:46.066628 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.066612 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56j8l\" (UniqueName: \"kubernetes.io/projected/aff3a44b-6cad-449e-8733-f85e9899ecc3-kube-api-access-56j8l\") pod \"cluster-monitoring-operator-884576c57-nqw98\" (UID: \"aff3a44b-6cad-449e-8733-f85e9899ecc3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nqw98" Oct 30 00:04:46.077260 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.077244 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mpg4t_de19ca9c-b22d-49a4-a5a6-a5f3a61bf892/serve-healthcheck-canary/0.log" Oct 30 00:04:46.138869 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.138817 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-modprobe-d\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.138869 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.138841 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-kubelet\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.138869 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.138858 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-sys\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.139096 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.138873 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acc82f37-fad2-424d-89d0-5e5876cb75de-metrics-tls\") pod \"dns-default-52kw4\" (UID: \"acc82f37-fad2-424d-89d0-5e5876cb75de\") " pod="openshift-dns/dns-default-52kw4" Oct 30 00:04:46.139096 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.138897 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-var-lib-cni-bin\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.139096 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.138934 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-multus-conf-dir\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.139096 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.138960 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs\") pod \"network-metrics-daemon-52qpr\" (UID: \"30673bf9-d332-4592-afce-b16db49876cb\") " pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:46.139096 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.138983 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0440635e-2cf5-4f40-a4f5-e716376044ba-cnibin\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:46.139096 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.138982 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-sys\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.139096 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.138996 5001 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:04:46.139096 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139005 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-metrics-certs\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:46.139096 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139003 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-multus-conf-dir\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.139096 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139033 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-tls\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:46.139096 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.138981 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-kubelet\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.139096 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139042 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0440635e-2cf5-4f40-a4f5-e716376044ba-cnibin\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:46.139096 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.139057 5001 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 00:04:46.139096 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.138994 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-modprobe-d\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.139096 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139058 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-sysctl-d\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.139096 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.139075 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acc82f37-fad2-424d-89d0-5e5876cb75de-metrics-tls podName:acc82f37-fad2-424d-89d0-5e5876cb75de nodeName:}" failed. No retries permitted until 2026-10-30 00:04:46.639056691 +0000 UTC m=+1.844557873 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acc82f37-fad2-424d-89d0-5e5876cb75de-metrics-tls") pod "dns-default-52kw4" (UID: "acc82f37-fad2-424d-89d0-5e5876cb75de") : secret "dns-default-metrics-tls" not found Oct 30 00:04:46.139096 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.139098 5001 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.139127 5001 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.138994 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-var-lib-cni-bin\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.139141 5001 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c8c8757f-tspqd: secret "image-registry-tls" not found Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139143 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-lib-modules\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.139157 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-metrics-certs podName:9d080b8e-cca6-4040-830e-cc9e9e25c50a nodeName:}" failed. No retries permitted until 2026-10-30 00:04:46.639141571 +0000 UTC m=+1.844642744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-metrics-certs") pod "router-default-645c8fd54c-hp29d" (UID: "9d080b8e-cca6-4040-830e-cc9e9e25c50a") : secret "router-metrics-certs-default" not found Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139172 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-sysctl-d\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.139183 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs podName:30673bf9-d332-4592-afce-b16db49876cb nodeName:}" failed. No retries permitted until 2026-10-30 00:04:46.639176456 +0000 UTC m=+1.844677619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs") pod "network-metrics-daemon-52qpr" (UID: "30673bf9-d332-4592-afce-b16db49876cb") : secret "metrics-daemon-secret" not found Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.139207 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-tls podName:fd8f9a4a-5b77-4a52-9ea9-a12245c00489 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:46.639199877 +0000 UTC m=+1.844701039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-tls") pod "image-registry-7c8c8757f-tspqd" (UID: "fd8f9a4a-5b77-4a52-9ea9-a12245c00489") : secret "image-registry-tls" not found Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139224 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10514d09-1dbf-47c2-bc1b-54a15f384cb3-host-slash\") pod \"iptables-alerter-j6chh\" (UID: \"10514d09-1dbf-47c2-bc1b-54a15f384cb3\") " pod="openshift-network-operator/iptables-alerter-j6chh" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139225 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-lib-modules\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139245 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139264 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-etc-kubernetes\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139268 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10514d09-1dbf-47c2-bc1b-54a15f384cb3-host-slash\") pod \"iptables-alerter-j6chh\" (UID: \"10514d09-1dbf-47c2-bc1b-54a15f384cb3\") " pod="openshift-network-operator/iptables-alerter-j6chh" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139279 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139283 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139309 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-socket-dir\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139335 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb8917db-c1ba-43af-995e-1948de69aa29-dbus\") pod \"global-pull-secret-syncer-zm28x\" (UID: \"fb8917db-c1ba-43af-995e-1948de69aa29\") " pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139339 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139310 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-etc-kubernetes\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139363 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-socket-dir\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139373 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/12aceb3a-2476-4efc-94d8-e6ec78cfc680-crio-socket\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139395 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb8917db-c1ba-43af-995e-1948de69aa29-dbus\") pod \"global-pull-secret-syncer-zm28x\" (UID: \"fb8917db-c1ba-43af-995e-1948de69aa29\") " pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139406 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-sysconfig\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139424 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-kubernetes\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139444 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/12aceb3a-2476-4efc-94d8-e6ec78cfc680-crio-socket\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139450 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-run-netns\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139465 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-sysconfig\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139474 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-cni-netd\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139479 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-kubernetes\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139498 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-sys-fs\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139505 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-run-netns\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139536 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-cnibin\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139547 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-cni-netd\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139554 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-sys-fs\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139554 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-multus-socket-dir-parent\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139583 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-host\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139594 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-cnibin\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139599 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-systemd\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139622 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-sysctl-conf\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.139606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139627 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-systemd\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139643 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-run-netns\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139661 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-multus-socket-dir-parent\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139663 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-host\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139668 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-run-openvswitch\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139683 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-run-netns\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139694 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l25ts\" (UID: \"18a4ebbe-c390-49d8-99e7-858bfaa3b834\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l25ts" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139700 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-run-openvswitch\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139718 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0440635e-2cf5-4f40-a4f5-e716376044ba-system-cni-dir\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.139743 5001 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139753 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-etc-sysctl-conf\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139740 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-run\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.139772 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-tls podName:18a4ebbe-c390-49d8-99e7-858bfaa3b834 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:46.63976332 +0000 UTC m=+1.845264482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-l25ts" (UID: "18a4ebbe-c390-49d8-99e7-858bfaa3b834") : secret "samples-operator-tls" not found Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139796 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0440635e-2cf5-4f40-a4f5-e716376044ba-system-cni-dir\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139803 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0440635e-2cf5-4f40-a4f5-e716376044ba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139829 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-run-k8s-cni-cncf-io\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139799 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-run\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139870 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-run-k8s-cni-cncf-io\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139844 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d080b8e-cca6-4040-830e-cc9e9e25c50a-service-ca-bundle\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139892 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0440635e-2cf5-4f40-a4f5-e716376044ba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l54xc\" (UID: \"0440635e-2cf5-4f40-a4f5-e716376044ba\") " pod="openshift-multus/multus-additional-cni-plugins-l54xc" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139921 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-cni-bin\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139940 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-multus-cni-dir\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139942 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-cni-bin\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.139966 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d080b8e-cca6-4040-830e-cc9e9e25c50a-service-ca-bundle podName:9d080b8e-cca6-4040-830e-cc9e9e25c50a nodeName:}" failed. No retries permitted until 2026-10-30 00:04:46.639949858 +0000 UTC m=+1.845451036 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9d080b8e-cca6-4040-830e-cc9e9e25c50a-service-ca-bundle") pod "router-default-645c8fd54c-hp29d" (UID: "9d080b8e-cca6-4040-830e-cc9e9e25c50a") : configmap references non-existent config key: service-ca.crt Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.139980 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-multus-cni-dir\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140009 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-var-lib-kubelet\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140025 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-etc-openvswitch\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140040 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-var-lib-kubelet\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140042 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-log-socket\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140058 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-log-socket\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140069 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-run-systemd\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140081 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-etc-openvswitch\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140084 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-run-ovn-kubernetes\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140102 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-run-systemd\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140103 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-var-lib-cni-multus\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140120 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-var-lib-cni-multus\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140127 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-run-multus-certs\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140143 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-run-ovn-kubernetes\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140164 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-host-run-multus-certs\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140143 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12aceb3a-2476-4efc-94d8-e6ec78cfc680-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140185 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-run-ovn\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.140189 5001 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140216 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-run-ovn\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140210 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7m2x2\" (UID: \"0d3be8a7-773b-42e1-a9a1-60d7df717301\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7m2x2" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.140219 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12aceb3a-2476-4efc-94d8-e6ec78cfc680-insights-runtime-extractor-tls podName:12aceb3a-2476-4efc-94d8-e6ec78cfc680 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:46.640210057 +0000 UTC m=+1.845711248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/12aceb3a-2476-4efc-94d8-e6ec78cfc680-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dd8wx" (UID: "12aceb3a-2476-4efc-94d8-e6ec78cfc680") : secret "insights-runtime-extractor-tls" not found Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.140258 5001 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140266 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12aceb3a-2476-4efc-94d8-e6ec78cfc680-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.140278 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert podName:0d3be8a7-773b-42e1-a9a1-60d7df717301 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:46.640272207 +0000 UTC m=+1.845773414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7m2x2" (UID: "0d3be8a7-773b-42e1-a9a1-60d7df717301") : secret "networking-console-plugin-cert" not found Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140294 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-etc-selinux\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140312 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-systemd-units\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140327 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-registry-tls\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140350 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-hostroot\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140368 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb8917db-c1ba-43af-995e-1948de69aa29-kubelet-config\") pod \"global-pull-secret-syncer-zm28x\" (UID: \"fb8917db-c1ba-43af-995e-1948de69aa29\") " pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.140390 5001 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140392 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-etc-selinux\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.140401 5001 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dcddbb874-9zx5v: secret "image-registry-tls" not found Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140398 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aff3a44b-6cad-449e-8733-f85e9899ecc3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nqw98\" (UID: \"aff3a44b-6cad-449e-8733-f85e9899ecc3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nqw98" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140410 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb8917db-c1ba-43af-995e-1948de69aa29-kubelet-config\") pod \"global-pull-secret-syncer-zm28x\" (UID: \"fb8917db-c1ba-43af-995e-1948de69aa29\") " pod="kube-system/global-pull-secret-syncer-zm28x" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140419 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-hostroot\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140426 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-device-dir\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140438 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-systemd-units\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.140442 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-registry-tls podName:526cc561-1bbd-4079-8a2f-326f939753e0 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:46.640429027 +0000 UTC m=+1.845930187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-registry-tls") pod "image-registry-7dcddbb874-9zx5v" (UID: "526cc561-1bbd-4079-8a2f-326f939753e0") : secret "image-registry-tls" not found Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.140468 5001 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140470 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-device-dir\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140474 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-registration-dir\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.140499 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff3a44b-6cad-449e-8733-f85e9899ecc3-cluster-monitoring-operator-tls podName:aff3a44b-6cad-449e-8733-f85e9899ecc3 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:46.640491608 +0000 UTC m=+1.845992767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/aff3a44b-6cad-449e-8733-f85e9899ecc3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nqw98" (UID: "aff3a44b-6cad-449e-8733-f85e9899ecc3") : secret "cluster-monitoring-operator-tls" not found Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140512 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85b08631-ff08-461f-8bda-6a64a609e1f1-host\") pod \"node-ca-qs54p\" (UID: \"85b08631-ff08-461f-8bda-6a64a609e1f1\") " pod="openshift-image-registry/node-ca-qs54p" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140517 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/31c2770e-628c-4bcd-a088-73f190c952cd-registration-dir\") pod \"aws-ebs-csi-driver-node-9db5f\" (UID: \"31c2770e-628c-4bcd-a088-73f190c952cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9db5f" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140536 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a9e7a0c8-43b3-4f8c-8870-505edace7f78-hosts-file\") pod \"node-resolver-b54v5\" (UID: \"a9e7a0c8-43b3-4f8c-8870-505edace7f78\") " pod="openshift-dns/node-resolver-b54v5" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140543 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85b08631-ff08-461f-8bda-6a64a609e1f1-host\") pod \"node-ca-qs54p\" (UID: \"85b08631-ff08-461f-8bda-6a64a609e1f1\") " pod="openshift-image-registry/node-ca-qs54p" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140557 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-var-lib-kubelet\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140562 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a9e7a0c8-43b3-4f8c-8870-505edace7f78-hosts-file\") pod \"node-resolver-b54v5\" (UID: \"a9e7a0c8-43b3-4f8c-8870-505edace7f78\") " pod="openshift-dns/node-resolver-b54v5" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140592 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-slash\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140603 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fed8e88-bc49-4564-abcc-7d0fff068aa4-var-lib-kubelet\") pod \"tuned-575b9\" (UID: \"0fed8e88-bc49-4564-abcc-7d0fff068aa4\") " pod="openshift-cluster-node-tuning-operator/tuned-575b9" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140611 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-var-lib-openvswitch\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140640 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-host-slash\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140650 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-node-log\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140671 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-var-lib-openvswitch\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140677 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-system-cni-dir\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140707 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27614fa1-66da-4d21-a0d1-0ac85a9bdd13-system-cni-dir\") pod \"multus-56lxt\" (UID: \"27614fa1-66da-4d21-a0d1-0ac85a9bdd13\") " pod="openshift-multus/multus-56lxt" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140681 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2277c280-7906-4c42-87f1-7941012a2d31-node-log\") pod \"ovnkube-node-7k49v\" (UID: \"2277c280-7906-4c42-87f1-7941012a2d31\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:46.141094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.140795 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12aceb3a-2476-4efc-94d8-e6ec78cfc680-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:46.219020 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.218993 5001 scope.go:122] "RemoveContainer" containerID="1b5598b379728a399ec94fb1bd83609608ffec6b569f58caae7897424bbb8d5f" Oct 30 00:04:46.646218 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.646131 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12aceb3a-2476-4efc-94d8-e6ec78cfc680-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:46.646218 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.646178 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7m2x2\" (UID: \"0d3be8a7-773b-42e1-a9a1-60d7df717301\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7m2x2" Oct 30 00:04:46.646218 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.646207 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-registry-tls\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:46.646509 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.646236 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aff3a44b-6cad-449e-8733-f85e9899ecc3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nqw98\" (UID: \"aff3a44b-6cad-449e-8733-f85e9899ecc3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nqw98" Oct 30 00:04:46.646509 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.646279 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acc82f37-fad2-424d-89d0-5e5876cb75de-metrics-tls\") pod \"dns-default-52kw4\" (UID: \"acc82f37-fad2-424d-89d0-5e5876cb75de\") " pod="openshift-dns/dns-default-52kw4" Oct 30 00:04:46.646509 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.646304 5001 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 00:04:46.646509 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.646309 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs\") pod \"network-metrics-daemon-52qpr\" (UID: \"30673bf9-d332-4592-afce-b16db49876cb\") " pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:46.646509 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.646304 5001 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:04:46.646509 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.646373 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert podName:0d3be8a7-773b-42e1-a9a1-60d7df717301 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:47.646353703 +0000 UTC m=+2.851854866 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7m2x2" (UID: "0d3be8a7-773b-42e1-a9a1-60d7df717301") : secret "networking-console-plugin-cert" not found Oct 30 00:04:46.646509 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.646376 5001 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 00:04:46.646509 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.646376 5001 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:04:46.646509 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.646396 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-metrics-certs\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:46.646509 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.646374 5001 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:04:46.646509 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.646375 5001 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 00:04:46.646509 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.646422 5001 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dcddbb874-9zx5v: secret "image-registry-tls" not found Oct 30 00:04:46.646509 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.646423 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12aceb3a-2476-4efc-94d8-e6ec78cfc680-insights-runtime-extractor-tls podName:12aceb3a-2476-4efc-94d8-e6ec78cfc680 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:47.646406943 +0000 UTC m=+2.851908103 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/12aceb3a-2476-4efc-94d8-e6ec78cfc680-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dd8wx" (UID: "12aceb3a-2476-4efc-94d8-e6ec78cfc680") : secret "insights-runtime-extractor-tls" not found Oct 30 00:04:46.646509 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.646446 5001 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 00:04:46.646509 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.646466 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs podName:30673bf9-d332-4592-afce-b16db49876cb nodeName:}" failed. No retries permitted until 2026-10-30 00:04:47.646453442 +0000 UTC m=+2.851954605 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs") pod "network-metrics-daemon-52qpr" (UID: "30673bf9-d332-4592-afce-b16db49876cb") : secret "metrics-daemon-secret" not found Oct 30 00:04:46.646509 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.646486 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-metrics-certs podName:9d080b8e-cca6-4040-830e-cc9e9e25c50a nodeName:}" failed. No retries permitted until 2026-10-30 00:04:47.646477215 +0000 UTC m=+2.851978389 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-metrics-certs") pod "router-default-645c8fd54c-hp29d" (UID: "9d080b8e-cca6-4040-830e-cc9e9e25c50a") : secret "router-metrics-certs-default" not found Oct 30 00:04:46.646509 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.646507 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acc82f37-fad2-424d-89d0-5e5876cb75de-metrics-tls podName:acc82f37-fad2-424d-89d0-5e5876cb75de nodeName:}" failed. No retries permitted until 2026-10-30 00:04:47.646502098 +0000 UTC m=+2.852003257 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acc82f37-fad2-424d-89d0-5e5876cb75de-metrics-tls") pod "dns-default-52kw4" (UID: "acc82f37-fad2-424d-89d0-5e5876cb75de") : secret "dns-default-metrics-tls" not found Oct 30 00:04:46.647009 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.646528 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff3a44b-6cad-449e-8733-f85e9899ecc3-cluster-monitoring-operator-tls podName:aff3a44b-6cad-449e-8733-f85e9899ecc3 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:47.646522462 +0000 UTC m=+2.852023621 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/aff3a44b-6cad-449e-8733-f85e9899ecc3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nqw98" (UID: "aff3a44b-6cad-449e-8733-f85e9899ecc3") : secret "cluster-monitoring-operator-tls" not found Oct 30 00:04:46.647009 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.646539 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-registry-tls podName:526cc561-1bbd-4079-8a2f-326f939753e0 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:47.646533316 +0000 UTC m=+2.852034475 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-registry-tls") pod "image-registry-7dcddbb874-9zx5v" (UID: "526cc561-1bbd-4079-8a2f-326f939753e0") : secret "image-registry-tls" not found Oct 30 00:04:46.647009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.646552 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-tls\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:46.647009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.646605 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l25ts\" (UID: \"18a4ebbe-c390-49d8-99e7-858bfaa3b834\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l25ts" Oct 30 00:04:46.647009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.646626 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d080b8e-cca6-4040-830e-cc9e9e25c50a-service-ca-bundle\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:46.647009 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.646699 5001 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:04:46.647009 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.646701 5001 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 00:04:46.647009 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.646728 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d080b8e-cca6-4040-830e-cc9e9e25c50a-service-ca-bundle podName:9d080b8e-cca6-4040-830e-cc9e9e25c50a nodeName:}" failed. No retries permitted until 2026-10-30 00:04:47.646719911 +0000 UTC m=+2.852221085 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9d080b8e-cca6-4040-830e-cc9e9e25c50a-service-ca-bundle") pod "router-default-645c8fd54c-hp29d" (UID: "9d080b8e-cca6-4040-830e-cc9e9e25c50a") : configmap references non-existent config key: service-ca.crt Oct 30 00:04:46.647009 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.646708 5001 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c8c8757f-tspqd: secret "image-registry-tls" not found Oct 30 00:04:46.647009 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.646740 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-tls podName:18a4ebbe-c390-49d8-99e7-858bfaa3b834 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:47.646735431 +0000 UTC m=+2.852236589 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-l25ts" (UID: "18a4ebbe-c390-49d8-99e7-858bfaa3b834") : secret "samples-operator-tls" not found Oct 30 00:04:46.647009 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.646780 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-tls podName:fd8f9a4a-5b77-4a52-9ea9-a12245c00489 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:47.646767947 +0000 UTC m=+2.852269109 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-tls") pod "image-registry-7c8c8757f-tspqd" (UID: "fd8f9a4a-5b77-4a52-9ea9-a12245c00489") : secret "image-registry-tls" not found Oct 30 00:04:46.878178 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.878151 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-xt8f2_8d95c49a-0861-4878-b10f-6b51c391b19e/migrator/0.log" Oct 30 00:04:46.970094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.970026 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8gb5_9fa688d8-29ca-4c1f-926a-f1a4099e6b44/console-operator/2.log" Oct 30 00:04:46.970449 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.970429 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8gb5_9fa688d8-29ca-4c1f-926a-f1a4099e6b44/console-operator/1.log" Oct 30 00:04:46.970573 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.970466 5001 generic.go:356] "Generic (PLEG): container finished" podID="9fa688d8-29ca-4c1f-926a-f1a4099e6b44" containerID="59b0f5a0e2f537c02e23f6186ca5e0632a9bd3a31006debee72226ac8ef94d75" exitCode=255 Oct 30 00:04:46.970573 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.970543 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" event={"ID":"9fa688d8-29ca-4c1f-926a-f1a4099e6b44","Type":"ContainerDied","Data":"59b0f5a0e2f537c02e23f6186ca5e0632a9bd3a31006debee72226ac8ef94d75"} Oct 30 00:04:46.970668 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.970603 5001 scope.go:122] "RemoveContainer" containerID="1b5598b379728a399ec94fb1bd83609608ffec6b569f58caae7897424bbb8d5f" Oct 30 00:04:46.971088 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:46.971071 5001 scope.go:122] "RemoveContainer" containerID="59b0f5a0e2f537c02e23f6186ca5e0632a9bd3a31006debee72226ac8ef94d75" Oct 30 00:04:46.971232 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:46.971215 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-z8gb5_openshift-console-operator(9fa688d8-29ca-4c1f-926a-f1a4099e6b44)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" podUID="9fa688d8-29ca-4c1f-926a-f1a4099e6b44" Oct 30 00:04:47.077864 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:47.077820 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l_c1a647f1-e715-4a8a-8b44-0196b3b1af27/kube-storage-version-migrator-operator/0.log" Oct 30 00:04:47.306449 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:47.306416 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sxvqx" Oct 30 00:04:47.658452 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:47.658368 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12aceb3a-2476-4efc-94d8-e6ec78cfc680-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:47.658452 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:47.658414 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7m2x2\" (UID: \"0d3be8a7-773b-42e1-a9a1-60d7df717301\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7m2x2" Oct 30 00:04:47.658652 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:47.658499 5001 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:04:47.658652 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:47.658537 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-registry-tls\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:47.658652 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:47.658552 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12aceb3a-2476-4efc-94d8-e6ec78cfc680-insights-runtime-extractor-tls podName:12aceb3a-2476-4efc-94d8-e6ec78cfc680 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:49.658538641 +0000 UTC m=+4.864039833 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/12aceb3a-2476-4efc-94d8-e6ec78cfc680-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dd8wx" (UID: "12aceb3a-2476-4efc-94d8-e6ec78cfc680") : secret "insights-runtime-extractor-tls" not found Oct 30 00:04:47.658652 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:47.658578 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aff3a44b-6cad-449e-8733-f85e9899ecc3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nqw98\" (UID: \"aff3a44b-6cad-449e-8733-f85e9899ecc3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nqw98" Oct 30 00:04:47.658652 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:47.658500 5001 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 00:04:47.658652 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:47.658618 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acc82f37-fad2-424d-89d0-5e5876cb75de-metrics-tls\") pod \"dns-default-52kw4\" (UID: \"acc82f37-fad2-424d-89d0-5e5876cb75de\") " pod="openshift-dns/dns-default-52kw4" Oct 30 00:04:47.658958 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:47.658657 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert podName:0d3be8a7-773b-42e1-a9a1-60d7df717301 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:49.658638041 +0000 UTC m=+4.864139215 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7m2x2" (UID: "0d3be8a7-773b-42e1-a9a1-60d7df717301") : secret "networking-console-plugin-cert" not found Oct 30 00:04:47.658958 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:47.658684 5001 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 00:04:47.658958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:47.658682 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs\") pod \"network-metrics-daemon-52qpr\" (UID: \"30673bf9-d332-4592-afce-b16db49876cb\") " pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:47.658958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:47.658718 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-metrics-certs\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:47.658958 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:47.658727 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff3a44b-6cad-449e-8733-f85e9899ecc3-cluster-monitoring-operator-tls podName:aff3a44b-6cad-449e-8733-f85e9899ecc3 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:49.658714321 +0000 UTC m=+4.864215500 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/aff3a44b-6cad-449e-8733-f85e9899ecc3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nqw98" (UID: "aff3a44b-6cad-449e-8733-f85e9899ecc3") : secret "cluster-monitoring-operator-tls" not found Oct 30 00:04:47.658958 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:47.658752 5001 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 00:04:47.658958 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:47.658684 5001 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:04:47.658958 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:47.658788 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs podName:30673bf9-d332-4592-afce-b16db49876cb nodeName:}" failed. No retries permitted until 2026-10-30 00:04:49.65877665 +0000 UTC m=+4.864277826 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs") pod "network-metrics-daemon-52qpr" (UID: "30673bf9-d332-4592-afce-b16db49876cb") : secret "metrics-daemon-secret" not found Oct 30 00:04:47.658958 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:47.658817 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acc82f37-fad2-424d-89d0-5e5876cb75de-metrics-tls podName:acc82f37-fad2-424d-89d0-5e5876cb75de nodeName:}" failed. No retries permitted until 2026-10-30 00:04:49.658797998 +0000 UTC m=+4.864299171 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acc82f37-fad2-424d-89d0-5e5876cb75de-metrics-tls") pod "dns-default-52kw4" (UID: "acc82f37-fad2-424d-89d0-5e5876cb75de") : secret "dns-default-metrics-tls" not found Oct 30 00:04:47.658958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:47.658761 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-tls\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:47.658958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:47.658825 5001 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 00:04:47.658958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:47.658930 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l25ts\" (UID: \"18a4ebbe-c390-49d8-99e7-858bfaa3b834\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l25ts" Oct 30 00:04:47.658958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:47.658960 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d080b8e-cca6-4040-830e-cc9e9e25c50a-service-ca-bundle\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:47.659613 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:47.659069 5001 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 00:04:47.659613 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:47.659123 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-tls podName:18a4ebbe-c390-49d8-99e7-858bfaa3b834 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:49.659106382 +0000 UTC m=+4.864607548 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-l25ts" (UID: "18a4ebbe-c390-49d8-99e7-858bfaa3b834") : secret "samples-operator-tls" not found Oct 30 00:04:47.659613 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:47.659144 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d080b8e-cca6-4040-830e-cc9e9e25c50a-service-ca-bundle podName:9d080b8e-cca6-4040-830e-cc9e9e25c50a nodeName:}" failed. No retries permitted until 2026-10-30 00:04:49.659131732 +0000 UTC m=+4.864632898 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9d080b8e-cca6-4040-830e-cc9e9e25c50a-service-ca-bundle") pod "router-default-645c8fd54c-hp29d" (UID: "9d080b8e-cca6-4040-830e-cc9e9e25c50a") : configmap references non-existent config key: service-ca.crt Oct 30 00:04:47.661068 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:47.661048 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-registry-tls\") pod \"image-registry-7dcddbb874-9zx5v\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:47.661184 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:47.661088 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-tls\") pod \"image-registry-7c8c8757f-tspqd\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:47.661184 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:47.661154 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d080b8e-cca6-4040-830e-cc9e9e25c50a-metrics-certs\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:47.718295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:47.718263 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:47.725738 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:47.725716 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:47.975838 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:47.975771 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8gb5_9fa688d8-29ca-4c1f-926a-f1a4099e6b44/console-operator/2.log" Oct 30 00:04:48.036060 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:48.035997 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c8c8757f-tspqd"] Oct 30 00:04:48.038793 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:48.038775 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7dcddbb874-9zx5v"] Oct 30 00:04:48.041405 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:48.041370 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd8f9a4a_5b77_4a52_9ea9_a12245c00489.slice/crio-fc6457817a7ac391f70b9adcd5eccc2c79736f72ff0b2947e3fd0c9cb43efe1a WatchSource:0}: Error finding container fc6457817a7ac391f70b9adcd5eccc2c79736f72ff0b2947e3fd0c9cb43efe1a: Status 404 returned error can't find the container with id fc6457817a7ac391f70b9adcd5eccc2c79736f72ff0b2947e3fd0c9cb43efe1a Oct 30 00:04:48.053875 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:48.053854 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod526cc561_1bbd_4079_8a2f_326f939753e0.slice/crio-06d9d93ac69d98dd8430cd691fecfc93a0fed308d287385ee18201c59866cd3a WatchSource:0}: Error finding container 06d9d93ac69d98dd8430cd691fecfc93a0fed308d287385ee18201c59866cd3a: Status 404 returned error can't find the container with id 06d9d93ac69d98dd8430cd691fecfc93a0fed308d287385ee18201c59866cd3a Oct 30 00:04:48.203805 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:48.203778 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:48.468318 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:48.468279 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" Oct 30 00:04:48.468653 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:48.468639 5001 scope.go:122] "RemoveContainer" containerID="59b0f5a0e2f537c02e23f6186ca5e0632a9bd3a31006debee72226ac8ef94d75" Oct 30 00:04:48.468838 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:48.468818 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-z8gb5_openshift-console-operator(9fa688d8-29ca-4c1f-926a-f1a4099e6b44)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" podUID="9fa688d8-29ca-4c1f-926a-f1a4099e6b44" Oct 30 00:04:48.983435 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:48.983398 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" event={"ID":"fd8f9a4a-5b77-4a52-9ea9-a12245c00489","Type":"ContainerStarted","Data":"1b8dd530cc65943016df08b487f918097c749098eda52357274e0e2ccc05ae86"} Oct 30 00:04:48.983902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:48.983446 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" event={"ID":"fd8f9a4a-5b77-4a52-9ea9-a12245c00489","Type":"ContainerStarted","Data":"fc6457817a7ac391f70b9adcd5eccc2c79736f72ff0b2947e3fd0c9cb43efe1a"} Oct 30 00:04:48.983902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:48.983486 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:04:48.985000 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:48.984976 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" event={"ID":"526cc561-1bbd-4079-8a2f-326f939753e0","Type":"ContainerStarted","Data":"aeaca81a572026cf1ae8ce575a28ae77ced4f7cc01bdee44413a58fb64cb0530"} Oct 30 00:04:48.985080 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:48.985009 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" event={"ID":"526cc561-1bbd-4079-8a2f-326f939753e0","Type":"ContainerStarted","Data":"06d9d93ac69d98dd8430cd691fecfc93a0fed308d287385ee18201c59866cd3a"} Oct 30 00:04:49.678231 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:49.678191 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aff3a44b-6cad-449e-8733-f85e9899ecc3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nqw98\" (UID: \"aff3a44b-6cad-449e-8733-f85e9899ecc3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nqw98" Oct 30 00:04:49.678386 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:49.678243 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acc82f37-fad2-424d-89d0-5e5876cb75de-metrics-tls\") pod \"dns-default-52kw4\" (UID: \"acc82f37-fad2-424d-89d0-5e5876cb75de\") " pod="openshift-dns/dns-default-52kw4" Oct 30 00:04:49.678386 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:49.678263 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs\") pod \"network-metrics-daemon-52qpr\" (UID: \"30673bf9-d332-4592-afce-b16db49876cb\") " pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:49.678386 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:49.678317 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l25ts\" (UID: \"18a4ebbe-c390-49d8-99e7-858bfaa3b834\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l25ts" Oct 30 00:04:49.678386 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:49.678338 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d080b8e-cca6-4040-830e-cc9e9e25c50a-service-ca-bundle\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:49.678386 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:49.678358 5001 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:04:49.678386 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:49.678381 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12aceb3a-2476-4efc-94d8-e6ec78cfc680-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:49.678701 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:49.678414 5001 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 00:04:49.678701 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:49.678446 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acc82f37-fad2-424d-89d0-5e5876cb75de-metrics-tls podName:acc82f37-fad2-424d-89d0-5e5876cb75de nodeName:}" failed. No retries permitted until 2026-10-30 00:04:53.678423722 +0000 UTC m=+8.883924895 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acc82f37-fad2-424d-89d0-5e5876cb75de-metrics-tls") pod "dns-default-52kw4" (UID: "acc82f37-fad2-424d-89d0-5e5876cb75de") : secret "dns-default-metrics-tls" not found Oct 30 00:04:49.678701 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:49.678465 5001 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 00:04:49.678701 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:49.678470 5001 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:04:49.678701 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:49.678488 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff3a44b-6cad-449e-8733-f85e9899ecc3-cluster-monitoring-operator-tls podName:aff3a44b-6cad-449e-8733-f85e9899ecc3 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:53.678469525 +0000 UTC m=+8.883970704 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/aff3a44b-6cad-449e-8733-f85e9899ecc3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nqw98" (UID: "aff3a44b-6cad-449e-8733-f85e9899ecc3") : secret "cluster-monitoring-operator-tls" not found Oct 30 00:04:49.678701 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:49.678412 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7m2x2\" (UID: \"0d3be8a7-773b-42e1-a9a1-60d7df717301\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7m2x2" Oct 30 00:04:49.678701 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:49.678420 5001 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 00:04:49.678701 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:49.678533 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12aceb3a-2476-4efc-94d8-e6ec78cfc680-insights-runtime-extractor-tls podName:12aceb3a-2476-4efc-94d8-e6ec78cfc680 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:53.678520039 +0000 UTC m=+8.884021211 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/12aceb3a-2476-4efc-94d8-e6ec78cfc680-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dd8wx" (UID: "12aceb3a-2476-4efc-94d8-e6ec78cfc680") : secret "insights-runtime-extractor-tls" not found Oct 30 00:04:49.678701 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:49.678551 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert podName:0d3be8a7-773b-42e1-a9a1-60d7df717301 nodeName:}" failed. No retries permitted until 2026-10-30 00:04:53.678541729 +0000 UTC m=+8.884042891 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7m2x2" (UID: "0d3be8a7-773b-42e1-a9a1-60d7df717301") : secret "networking-console-plugin-cert" not found Oct 30 00:04:49.678701 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:49.678568 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs podName:30673bf9-d332-4592-afce-b16db49876cb nodeName:}" failed. No retries permitted until 2026-10-30 00:04:53.678557318 +0000 UTC m=+8.884058477 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs") pod "network-metrics-daemon-52qpr" (UID: "30673bf9-d332-4592-afce-b16db49876cb") : secret "metrics-daemon-secret" not found Oct 30 00:04:49.679223 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:49.679138 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d080b8e-cca6-4040-830e-cc9e9e25c50a-service-ca-bundle\") pod \"router-default-645c8fd54c-hp29d\" (UID: \"9d080b8e-cca6-4040-830e-cc9e9e25c50a\") " pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:49.680683 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:49.680662 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/18a4ebbe-c390-49d8-99e7-858bfaa3b834-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l25ts\" (UID: \"18a4ebbe-c390-49d8-99e7-858bfaa3b834\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l25ts" Oct 30 00:04:49.817964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:49.817931 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:49.818984 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:49.818965 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l25ts" Oct 30 00:04:49.987799 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:49.987718 5001 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 00:04:50.115188 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:50.115159 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l25ts"] Oct 30 00:04:50.130506 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:50.130469 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-645c8fd54c-hp29d"] Oct 30 00:04:50.139061 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:50.139032 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d080b8e_cca6_4040_830e_cc9e9e25c50a.slice/crio-b9692813cdbbd8712ab9b51db34588ac11c656a4b33a7b095dea372f339b4a3d WatchSource:0}: Error finding container b9692813cdbbd8712ab9b51db34588ac11c656a4b33a7b095dea372f339b4a3d: Status 404 returned error can't find the container with id b9692813cdbbd8712ab9b51db34588ac11c656a4b33a7b095dea372f339b4a3d Oct 30 00:04:50.162613 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:50.162596 5001 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:04:50.992339 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:50.992302 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l25ts" event={"ID":"18a4ebbe-c390-49d8-99e7-858bfaa3b834","Type":"ContainerStarted","Data":"385848e0ef269105e7c016cd1adc8d0f76b5489cffc21ce26ea954ffe3416a71"} Oct 30 00:04:50.994251 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:50.994222 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-645c8fd54c-hp29d" event={"ID":"9d080b8e-cca6-4040-830e-cc9e9e25c50a","Type":"ContainerStarted","Data":"2dd6ac0c697bde2e45c9a0b5ef50628471a0efd7ea478bb0e746e834ee4b551a"} Oct 30 00:04:50.994352 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:50.994271 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-645c8fd54c-hp29d" event={"ID":"9d080b8e-cca6-4040-830e-cc9e9e25c50a","Type":"ContainerStarted","Data":"b9692813cdbbd8712ab9b51db34588ac11c656a4b33a7b095dea372f339b4a3d"} Oct 30 00:04:51.155212 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:51.155170 5001 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q8m2x_5166c4d7-2b82-4bbd-8d58-836cb274ed25/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:51.345788 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:51.345754 5001 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q8m2x_5166c4d7-2b82-4bbd-8d58-836cb274ed25/konnectivity-agent/0.log" limit=1024 Oct 30 00:04:51.819067 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:51.819030 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:51.822066 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:51.822042 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:51.968730 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:51.968688 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:51.970134 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:51.970109 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-645c8fd54c-hp29d" Oct 30 00:04:52.318643 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:52.318617 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:52.319007 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:52.318848 5001 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 00:04:53.002736 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:53.002698 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l25ts" event={"ID":"18a4ebbe-c390-49d8-99e7-858bfaa3b834","Type":"ContainerStarted","Data":"8612805eedfd4c995464bb2465ea1b739cedea8e07f040571325bf364574bb45"} Oct 30 00:04:53.002736 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:53.002754 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l25ts" event={"ID":"18a4ebbe-c390-49d8-99e7-858bfaa3b834","Type":"ContainerStarted","Data":"acbfe66391a0f087b68df6fd0e86031816853fc19ec38984af6a2aeb88b08c1b"} Oct 30 00:04:53.368905 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:53.368854 5001 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" Oct 30 00:04:53.369297 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:53.369283 5001 scope.go:122] "RemoveContainer" containerID="59b0f5a0e2f537c02e23f6186ca5e0632a9bd3a31006debee72226ac8ef94d75" Oct 30 00:04:53.369505 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:53.369484 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-z8gb5_openshift-console-operator(9fa688d8-29ca-4c1f-926a-f1a4099e6b44)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" podUID="9fa688d8-29ca-4c1f-926a-f1a4099e6b44" Oct 30 00:04:53.722797 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:53.722703 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12aceb3a-2476-4efc-94d8-e6ec78cfc680-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:04:53.722797 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:53.722743 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7m2x2\" (UID: \"0d3be8a7-773b-42e1-a9a1-60d7df717301\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7m2x2" Oct 30 00:04:53.722797 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:53.722767 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aff3a44b-6cad-449e-8733-f85e9899ecc3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nqw98\" (UID: \"aff3a44b-6cad-449e-8733-f85e9899ecc3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nqw98" Oct 30 00:04:53.723101 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:53.722799 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acc82f37-fad2-424d-89d0-5e5876cb75de-metrics-tls\") pod \"dns-default-52kw4\" (UID: \"acc82f37-fad2-424d-89d0-5e5876cb75de\") " pod="openshift-dns/dns-default-52kw4" Oct 30 00:04:53.723101 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:53.722819 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs\") pod \"network-metrics-daemon-52qpr\" (UID: \"30673bf9-d332-4592-afce-b16db49876cb\") " pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:04:53.723101 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:53.722837 5001 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:04:53.723101 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:53.722932 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12aceb3a-2476-4efc-94d8-e6ec78cfc680-insights-runtime-extractor-tls podName:12aceb3a-2476-4efc-94d8-e6ec78cfc680 nodeName:}" failed. No retries permitted until 2026-10-30 00:05:01.722896148 +0000 UTC m=+16.928397307 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/12aceb3a-2476-4efc-94d8-e6ec78cfc680-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dd8wx" (UID: "12aceb3a-2476-4efc-94d8-e6ec78cfc680") : secret "insights-runtime-extractor-tls" not found Oct 30 00:04:53.723101 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:53.722837 5001 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 00:04:53.723101 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:53.722954 5001 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 00:04:53.723101 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:53.723005 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert podName:0d3be8a7-773b-42e1-a9a1-60d7df717301 nodeName:}" failed. No retries permitted until 2026-10-30 00:05:01.722989975 +0000 UTC m=+16.928491154 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7m2x2" (UID: "0d3be8a7-773b-42e1-a9a1-60d7df717301") : secret "networking-console-plugin-cert" not found Oct 30 00:04:53.723101 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:04:53.723025 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs podName:30673bf9-d332-4592-afce-b16db49876cb nodeName:}" failed. No retries permitted until 2026-10-30 00:05:01.723014868 +0000 UTC m=+16.928516034 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs") pod "network-metrics-daemon-52qpr" (UID: "30673bf9-d332-4592-afce-b16db49876cb") : secret "metrics-daemon-secret" not found Oct 30 00:04:53.725108 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:53.725086 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acc82f37-fad2-424d-89d0-5e5876cb75de-metrics-tls\") pod \"dns-default-52kw4\" (UID: \"acc82f37-fad2-424d-89d0-5e5876cb75de\") " pod="openshift-dns/dns-default-52kw4" Oct 30 00:04:53.725252 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:53.725226 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aff3a44b-6cad-449e-8733-f85e9899ecc3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nqw98\" (UID: \"aff3a44b-6cad-449e-8733-f85e9899ecc3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nqw98" Oct 30 00:04:53.727075 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:53.727052 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-52kw4" Oct 30 00:04:54.016401 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:54.016371 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-52kw4"] Oct 30 00:04:54.019636 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:54.019609 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nqw98" Oct 30 00:04:54.021385 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:54.021359 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc82f37_fad2_424d_89d0_5e5876cb75de.slice/crio-c8b7171d3b6dccfafccdd371c1494f499699db1d7bade99def94da93c4ebbd63 WatchSource:0}: Error finding container c8b7171d3b6dccfafccdd371c1494f499699db1d7bade99def94da93c4ebbd63: Status 404 returned error can't find the container with id c8b7171d3b6dccfafccdd371c1494f499699db1d7bade99def94da93c4ebbd63 Oct 30 00:04:54.312973 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:54.312888 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-nqw98"] Oct 30 00:04:54.318061 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:04:54.318036 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff3a44b_6cad_449e_8733_f85e9899ecc3.slice/crio-c850bbbb172265784fa2802ac96856d4577e78214f8ded4700c005b158e8cbc6 WatchSource:0}: Error finding container c850bbbb172265784fa2802ac96856d4577e78214f8ded4700c005b158e8cbc6: Status 404 returned error can't find the container with id c850bbbb172265784fa2802ac96856d4577e78214f8ded4700c005b158e8cbc6 Oct 30 00:04:55.012710 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:55.012670 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nqw98" event={"ID":"aff3a44b-6cad-449e-8733-f85e9899ecc3","Type":"ContainerStarted","Data":"c850bbbb172265784fa2802ac96856d4577e78214f8ded4700c005b158e8cbc6"} Oct 30 00:04:55.014202 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:55.014175 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-52kw4" event={"ID":"acc82f37-fad2-424d-89d0-5e5876cb75de","Type":"ContainerStarted","Data":"c8b7171d3b6dccfafccdd371c1494f499699db1d7bade99def94da93c4ebbd63"} Oct 30 00:04:55.939022 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:55.937930 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7k49v" Oct 30 00:04:56.013337 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:56.013316 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:04:56.018785 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:56.018755 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-52kw4" event={"ID":"acc82f37-fad2-424d-89d0-5e5876cb75de","Type":"ContainerStarted","Data":"f8c87f81be10f4d399af22493cfc43aed40893b4de5b9824d61ebde14139e8bf"} Oct 30 00:04:56.018888 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:56.018810 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-52kw4" event={"ID":"acc82f37-fad2-424d-89d0-5e5876cb75de","Type":"ContainerStarted","Data":"f6008285938e06c6b7e2defd61764603343252b2648a101924afb48228ac4fb2"} Oct 30 00:04:56.018888 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:56.018876 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-52kw4" Oct 30 00:04:56.663336 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:56.663308 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-q8m2x" Oct 30 00:04:56.916177 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:56.916112 5001 scope.go:122] "RemoveContainer" containerID="59b0f5a0e2f537c02e23f6186ca5e0632a9bd3a31006debee72226ac8ef94d75" Oct 30 00:04:57.024054 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:57.024026 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8gb5_9fa688d8-29ca-4c1f-926a-f1a4099e6b44/console-operator/2.log" Oct 30 00:04:57.024434 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:57.024141 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" event={"ID":"9fa688d8-29ca-4c1f-926a-f1a4099e6b44","Type":"ContainerStarted","Data":"739c4f6273c70e01f6e0e4f8f4b389fc32abf6d5f262ce1a4f742e9eb27c7588"} Oct 30 00:04:57.024516 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:57.024447 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" Oct 30 00:04:57.025427 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:57.025396 5001 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-z8gb5 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.18:8443/readyz\": dial tcp 10.132.0.18:8443: connect: connection refused" start-of-body= Oct 30 00:04:57.025561 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:57.025447 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" podUID="9fa688d8-29ca-4c1f-926a-f1a4099e6b44" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.18:8443/readyz\": dial tcp 10.132.0.18:8443: connect: connection refused" Oct 30 00:04:57.025793 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:57.025773 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nqw98" event={"ID":"aff3a44b-6cad-449e-8733-f85e9899ecc3","Type":"ContainerStarted","Data":"84a8a771ac6b2de5ae1e3d969e22fc1f586c02bf86f7ae54a67aadd834810620"} Oct 30 00:04:58.034285 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:04:58.034256 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-z8gb5" Oct 30 00:05:00.276658 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:00.276627 5001 patch_prober.go:29] interesting pod/image-registry-7dcddbb874-9zx5v container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 00:05:00.277020 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:00.276674 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" podUID="526cc561-1bbd-4079-8a2f-326f939753e0" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:05:01.805430 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:01.805389 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12aceb3a-2476-4efc-94d8-e6ec78cfc680-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:05:01.805430 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:01.805433 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7m2x2\" (UID: \"0d3be8a7-773b-42e1-a9a1-60d7df717301\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7m2x2" Oct 30 00:05:01.805974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:01.805470 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs\") pod \"network-metrics-daemon-52qpr\" (UID: \"30673bf9-d332-4592-afce-b16db49876cb\") " pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:05:01.808148 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:01.808122 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30673bf9-d332-4592-afce-b16db49876cb-metrics-certs\") pod \"network-metrics-daemon-52qpr\" (UID: \"30673bf9-d332-4592-afce-b16db49876cb\") " pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:05:01.808214 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:01.808145 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12aceb3a-2476-4efc-94d8-e6ec78cfc680-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dd8wx\" (UID: \"12aceb3a-2476-4efc-94d8-e6ec78cfc680\") " pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:05:01.808214 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:01.808204 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d3be8a7-773b-42e1-a9a1-60d7df717301-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7m2x2\" (UID: \"0d3be8a7-773b-42e1-a9a1-60d7df717301\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7m2x2" Oct 30 00:05:01.820162 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:01.820134 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-7m2x2" Oct 30 00:05:01.825375 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:01.825352 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52qpr" Oct 30 00:05:01.829276 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:01.829255 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dd8wx" Oct 30 00:05:02.122997 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:02.122905 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-52qpr"] Oct 30 00:05:02.129614 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:05:02.129591 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30673bf9_d332_4592_afce_b16db49876cb.slice/crio-7fc8f388cc6c420c4e108444bad3973a37581738d83ccadffd0da891f89eef12 WatchSource:0}: Error finding container 7fc8f388cc6c420c4e108444bad3973a37581738d83ccadffd0da891f89eef12: Status 404 returned error can't find the container with id 7fc8f388cc6c420c4e108444bad3973a37581738d83ccadffd0da891f89eef12 Oct 30 00:05:02.135569 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:02.135546 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-7m2x2"] Oct 30 00:05:02.140677 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:05:02.140654 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d3be8a7_773b_42e1_a9a1_60d7df717301.slice/crio-36b4a406785b9cd0a5c1ca5a9a4cfb252a058289db1c3a68af3224ff0b7c6d47 WatchSource:0}: Error finding container 36b4a406785b9cd0a5c1ca5a9a4cfb252a058289db1c3a68af3224ff0b7c6d47: Status 404 returned error can't find the container with id 36b4a406785b9cd0a5c1ca5a9a4cfb252a058289db1c3a68af3224ff0b7c6d47 Oct 30 00:05:02.143624 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:05:02.143599 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12aceb3a_2476_4efc_94d8_e6ec78cfc680.slice/crio-fd3564be4683a3f28a961720328b9fa90651563c58c0dbb9e03229abf077c4db WatchSource:0}: Error finding container fd3564be4683a3f28a961720328b9fa90651563c58c0dbb9e03229abf077c4db: Status 404 returned error can't find the container with id fd3564be4683a3f28a961720328b9fa90651563c58c0dbb9e03229abf077c4db Oct 30 00:05:02.145443 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:02.145418 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dd8wx"] Oct 30 00:05:02.837993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:02.837507 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c8c8757f-tspqd"] Oct 30 00:05:03.048857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:03.048816 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-52qpr" event={"ID":"30673bf9-d332-4592-afce-b16db49876cb","Type":"ContainerStarted","Data":"7fc8f388cc6c420c4e108444bad3973a37581738d83ccadffd0da891f89eef12"} Oct 30 00:05:03.050375 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:03.050348 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-7m2x2" event={"ID":"0d3be8a7-773b-42e1-a9a1-60d7df717301","Type":"ContainerStarted","Data":"36b4a406785b9cd0a5c1ca5a9a4cfb252a058289db1c3a68af3224ff0b7c6d47"} Oct 30 00:05:03.052380 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:03.052349 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dd8wx" event={"ID":"12aceb3a-2476-4efc-94d8-e6ec78cfc680","Type":"ContainerStarted","Data":"08eb1a10a14f90c20d68865d53ffdcd427487cef3aaea06444266d3bc91ceadb"} Oct 30 00:05:03.052476 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:03.052396 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dd8wx" event={"ID":"12aceb3a-2476-4efc-94d8-e6ec78cfc680","Type":"ContainerStarted","Data":"c86acbac9e87bc1cd99c29bed43cdad73ebdb09360a4f6610ccea054a72e32aa"} Oct 30 00:05:03.052476 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:03.052415 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dd8wx" event={"ID":"12aceb3a-2476-4efc-94d8-e6ec78cfc680","Type":"ContainerStarted","Data":"fd3564be4683a3f28a961720328b9fa90651563c58c0dbb9e03229abf077c4db"} Oct 30 00:05:04.059191 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:04.059151 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-52qpr" event={"ID":"30673bf9-d332-4592-afce-b16db49876cb","Type":"ContainerStarted","Data":"35644fe4498b5fa1c79a015cc41868ed85bb59f1db655996533754bd320dbd25"} Oct 30 00:05:04.059685 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:04.059220 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-52qpr" event={"ID":"30673bf9-d332-4592-afce-b16db49876cb","Type":"ContainerStarted","Data":"331137e705eee32330496c285611130bafd255c880bd8830100968e495231592"} Oct 30 00:05:04.060831 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:04.060798 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-7m2x2" event={"ID":"0d3be8a7-773b-42e1-a9a1-60d7df717301","Type":"ContainerStarted","Data":"0993e5e63d69df7f5637caa5a84cb07b82bc01939fcd6f0b44339df3ca2b1ec6"} Oct 30 00:05:05.066478 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:05.066442 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dd8wx" event={"ID":"12aceb3a-2476-4efc-94d8-e6ec78cfc680","Type":"ContainerStarted","Data":"c91cf75efdbbfe9149327c61460d690953fe4763fd62647b4b2156f22327b5aa"} Oct 30 00:05:06.018371 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:06.018330 5001 patch_prober.go:29] interesting pod/image-registry-7dcddbb874-9zx5v container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 00:05:06.018568 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:06.018401 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" podUID="526cc561-1bbd-4079-8a2f-326f939753e0" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:05:06.028134 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:06.028111 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-52kw4" Oct 30 00:05:11.008702 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.008671 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qdn9n"] Oct 30 00:05:11.052253 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.052220 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qdn9n" Oct 30 00:05:11.108387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.108362 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 00:05:11.108526 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.108469 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 00:05:11.164486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.164459 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-x8k7p\"" Oct 30 00:05:11.164486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.164461 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 00:05:11.164697 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.164495 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 00:05:11.182888 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.182867 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xvbc\" (UniqueName: \"kubernetes.io/projected/a1b58bee-c614-4dbf-bf3c-3593b3da5069-kube-api-access-5xvbc\") pod \"node-exporter-qdn9n\" (UID: \"a1b58bee-c614-4dbf-bf3c-3593b3da5069\") " pod="openshift-monitoring/node-exporter-qdn9n" Oct 30 00:05:11.183024 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.182899 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1b58bee-c614-4dbf-bf3c-3593b3da5069-sys\") pod \"node-exporter-qdn9n\" (UID: \"a1b58bee-c614-4dbf-bf3c-3593b3da5069\") " pod="openshift-monitoring/node-exporter-qdn9n" Oct 30 00:05:11.183024 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.182939 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1b58bee-c614-4dbf-bf3c-3593b3da5069-node-exporter-textfile\") pod \"node-exporter-qdn9n\" (UID: \"a1b58bee-c614-4dbf-bf3c-3593b3da5069\") " pod="openshift-monitoring/node-exporter-qdn9n" Oct 30 00:05:11.183093 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.183021 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1b58bee-c614-4dbf-bf3c-3593b3da5069-metrics-client-ca\") pod \"node-exporter-qdn9n\" (UID: \"a1b58bee-c614-4dbf-bf3c-3593b3da5069\") " pod="openshift-monitoring/node-exporter-qdn9n" Oct 30 00:05:11.183093 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.183059 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1b58bee-c614-4dbf-bf3c-3593b3da5069-node-exporter-accelerators-collector-config\") pod \"node-exporter-qdn9n\" (UID: \"a1b58bee-c614-4dbf-bf3c-3593b3da5069\") " pod="openshift-monitoring/node-exporter-qdn9n" Oct 30 00:05:11.183161 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.183096 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1b58bee-c614-4dbf-bf3c-3593b3da5069-node-exporter-tls\") pod \"node-exporter-qdn9n\" (UID: \"a1b58bee-c614-4dbf-bf3c-3593b3da5069\") " pod="openshift-monitoring/node-exporter-qdn9n" Oct 30 00:05:11.183161 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.183114 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1b58bee-c614-4dbf-bf3c-3593b3da5069-node-exporter-wtmp\") pod \"node-exporter-qdn9n\" (UID: \"a1b58bee-c614-4dbf-bf3c-3593b3da5069\") " pod="openshift-monitoring/node-exporter-qdn9n" Oct 30 00:05:11.183161 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.183133 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1b58bee-c614-4dbf-bf3c-3593b3da5069-root\") pod \"node-exporter-qdn9n\" (UID: \"a1b58bee-c614-4dbf-bf3c-3593b3da5069\") " pod="openshift-monitoring/node-exporter-qdn9n" Oct 30 00:05:11.183245 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.183180 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1b58bee-c614-4dbf-bf3c-3593b3da5069-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qdn9n\" (UID: \"a1b58bee-c614-4dbf-bf3c-3593b3da5069\") " pod="openshift-monitoring/node-exporter-qdn9n" Oct 30 00:05:11.284556 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.284526 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1b58bee-c614-4dbf-bf3c-3593b3da5069-sys\") pod \"node-exporter-qdn9n\" (UID: \"a1b58bee-c614-4dbf-bf3c-3593b3da5069\") " pod="openshift-monitoring/node-exporter-qdn9n" Oct 30 00:05:11.284711 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.284564 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1b58bee-c614-4dbf-bf3c-3593b3da5069-node-exporter-textfile\") pod \"node-exporter-qdn9n\" (UID: \"a1b58bee-c614-4dbf-bf3c-3593b3da5069\") " pod="openshift-monitoring/node-exporter-qdn9n" Oct 30 00:05:11.284711 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.284584 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1b58bee-c614-4dbf-bf3c-3593b3da5069-metrics-client-ca\") pod \"node-exporter-qdn9n\" (UID: \"a1b58bee-c614-4dbf-bf3c-3593b3da5069\") " pod="openshift-monitoring/node-exporter-qdn9n" Oct 30 00:05:11.284711 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.284608 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1b58bee-c614-4dbf-bf3c-3593b3da5069-node-exporter-accelerators-collector-config\") pod \"node-exporter-qdn9n\" (UID: \"a1b58bee-c614-4dbf-bf3c-3593b3da5069\") " pod="openshift-monitoring/node-exporter-qdn9n" Oct 30 00:05:11.284711 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.284653 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1b58bee-c614-4dbf-bf3c-3593b3da5069-sys\") pod \"node-exporter-qdn9n\" (UID: \"a1b58bee-c614-4dbf-bf3c-3593b3da5069\") " pod="openshift-monitoring/node-exporter-qdn9n" Oct 30 00:05:11.284711 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.284660 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1b58bee-c614-4dbf-bf3c-3593b3da5069-node-exporter-tls\") pod \"node-exporter-qdn9n\" (UID: \"a1b58bee-c614-4dbf-bf3c-3593b3da5069\") " pod="openshift-monitoring/node-exporter-qdn9n" Oct 30 00:05:11.284955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.284738 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1b58bee-c614-4dbf-bf3c-3593b3da5069-node-exporter-wtmp\") pod \"node-exporter-qdn9n\" (UID: \"a1b58bee-c614-4dbf-bf3c-3593b3da5069\") " pod="openshift-monitoring/node-exporter-qdn9n" Oct 30 00:05:11.284955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.284767 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1b58bee-c614-4dbf-bf3c-3593b3da5069-root\") pod \"node-exporter-qdn9n\" (UID: \"a1b58bee-c614-4dbf-bf3c-3593b3da5069\") " pod="openshift-monitoring/node-exporter-qdn9n" Oct 30 00:05:11.284955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.284791 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1b58bee-c614-4dbf-bf3c-3593b3da5069-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qdn9n\" (UID: \"a1b58bee-c614-4dbf-bf3c-3593b3da5069\") " pod="openshift-monitoring/node-exporter-qdn9n" Oct 30 00:05:11.284955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.284852 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xvbc\" (UniqueName: \"kubernetes.io/projected/a1b58bee-c614-4dbf-bf3c-3593b3da5069-kube-api-access-5xvbc\") pod \"node-exporter-qdn9n\" (UID: \"a1b58bee-c614-4dbf-bf3c-3593b3da5069\") " pod="openshift-monitoring/node-exporter-qdn9n" Oct 30 00:05:11.284955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.284888 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1b58bee-c614-4dbf-bf3c-3593b3da5069-root\") pod \"node-exporter-qdn9n\" (UID: \"a1b58bee-c614-4dbf-bf3c-3593b3da5069\") " pod="openshift-monitoring/node-exporter-qdn9n" Oct 30 00:05:11.284955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.284947 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1b58bee-c614-4dbf-bf3c-3593b3da5069-node-exporter-wtmp\") pod \"node-exporter-qdn9n\" (UID: \"a1b58bee-c614-4dbf-bf3c-3593b3da5069\") " pod="openshift-monitoring/node-exporter-qdn9n" Oct 30 00:05:11.285254 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.285002 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1b58bee-c614-4dbf-bf3c-3593b3da5069-node-exporter-textfile\") pod \"node-exporter-qdn9n\" (UID: \"a1b58bee-c614-4dbf-bf3c-3593b3da5069\") " pod="openshift-monitoring/node-exporter-qdn9n" Oct 30 00:05:11.285303 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.285260 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1b58bee-c614-4dbf-bf3c-3593b3da5069-node-exporter-accelerators-collector-config\") pod \"node-exporter-qdn9n\" (UID: \"a1b58bee-c614-4dbf-bf3c-3593b3da5069\") " pod="openshift-monitoring/node-exporter-qdn9n" Oct 30 00:05:11.285303 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.285271 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1b58bee-c614-4dbf-bf3c-3593b3da5069-metrics-client-ca\") pod \"node-exporter-qdn9n\" (UID: \"a1b58bee-c614-4dbf-bf3c-3593b3da5069\") " pod="openshift-monitoring/node-exporter-qdn9n" Oct 30 00:05:11.287016 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.286998 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1b58bee-c614-4dbf-bf3c-3593b3da5069-node-exporter-tls\") pod \"node-exporter-qdn9n\" (UID: \"a1b58bee-c614-4dbf-bf3c-3593b3da5069\") " pod="openshift-monitoring/node-exporter-qdn9n" Oct 30 00:05:11.287126 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.287108 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1b58bee-c614-4dbf-bf3c-3593b3da5069-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qdn9n\" (UID: \"a1b58bee-c614-4dbf-bf3c-3593b3da5069\") " pod="openshift-monitoring/node-exporter-qdn9n" Oct 30 00:05:11.310195 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.310160 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xvbc\" (UniqueName: \"kubernetes.io/projected/a1b58bee-c614-4dbf-bf3c-3593b3da5069-kube-api-access-5xvbc\") pod \"node-exporter-qdn9n\" (UID: \"a1b58bee-c614-4dbf-bf3c-3593b3da5069\") " pod="openshift-monitoring/node-exporter-qdn9n" Oct 30 00:05:11.402498 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:11.402473 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qdn9n" Oct 30 00:05:11.434830 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:05:11.434795 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b58bee_c614_4dbf_bf3c_3593b3da5069.slice/crio-ad79863f1f1781598bb9c90f1d6bf8504c1b9907c7cafcf89c7077935253e4c7 WatchSource:0}: Error finding container ad79863f1f1781598bb9c90f1d6bf8504c1b9907c7cafcf89c7077935253e4c7: Status 404 returned error can't find the container with id ad79863f1f1781598bb9c90f1d6bf8504c1b9907c7cafcf89c7077935253e4c7 Oct 30 00:05:12.097437 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:12.097397 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qdn9n" event={"ID":"a1b58bee-c614-4dbf-bf3c-3593b3da5069","Type":"ContainerStarted","Data":"ad79863f1f1781598bb9c90f1d6bf8504c1b9907c7cafcf89c7077935253e4c7"} Oct 30 00:05:12.842257 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:12.842236 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:05:13.103144 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:13.103070 5001 generic.go:356] "Generic (PLEG): container finished" podID="a1b58bee-c614-4dbf-bf3c-3593b3da5069" containerID="2adf8ce1538b8525d983d1ac6f6f883c66acf3e47c6390a67a0572eebe14f99f" exitCode=0 Oct 30 00:05:13.103497 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:13.103136 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qdn9n" event={"ID":"a1b58bee-c614-4dbf-bf3c-3593b3da5069","Type":"ContainerDied","Data":"2adf8ce1538b8525d983d1ac6f6f883c66acf3e47c6390a67a0572eebe14f99f"} Oct 30 00:05:14.111395 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:14.111360 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qdn9n" event={"ID":"a1b58bee-c614-4dbf-bf3c-3593b3da5069","Type":"ContainerStarted","Data":"f4e21a3642dedff5e7f8871c8cf7737ac30d65fcaef293e90dc2665633f93db5"} Oct 30 00:05:14.111896 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:14.111428 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qdn9n" event={"ID":"a1b58bee-c614-4dbf-bf3c-3593b3da5069","Type":"ContainerStarted","Data":"9f26c8c8641ed174360d9f494ec04a099c29e6a06757ea494a06eedd7cd6254e"} Oct 30 00:05:14.175147 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:14.175088 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qdn9n" podStartSLOduration=2.72128745 podStartE2EDuration="4.17507602s" totalImagesPullingTime="1.372477711s" totalInitContainerRuntime="81.310859ms" isStatefulPod=true podCreationTimestamp="2026-10-30 00:05:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:05:14.165284923 +0000 UTC m=+29.370786103" watchObservedRunningTime="2026-10-30 00:05:14.17507602 +0000 UTC m=+29.380577198" Oct 30 00:05:16.018378 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:16.018352 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:05:17.309741 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:17.309711 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-sxvqx" Oct 30 00:05:17.322308 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:05:17.322272 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:05:24.786297 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:24.786266 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7dcddbb874-9zx5v"] Oct 30 00:05:27.872031 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:27.871941 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" podUID="fd8f9a4a-5b77-4a52-9ea9-a12245c00489" containerName="registry" containerID="cri-o://1b8dd530cc65943016df08b487f918097c749098eda52357274e0e2ccc05ae86" gracePeriod=30 Oct 30 00:05:28.112178 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.112150 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:05:28.165094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.165001 5001 generic.go:356] "Generic (PLEG): container finished" podID="fd8f9a4a-5b77-4a52-9ea9-a12245c00489" containerID="1b8dd530cc65943016df08b487f918097c749098eda52357274e0e2ccc05ae86" exitCode=0 Oct 30 00:05:28.165094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.165045 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" Oct 30 00:05:28.165314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.165090 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" event={"ID":"fd8f9a4a-5b77-4a52-9ea9-a12245c00489","Type":"ContainerDied","Data":"1b8dd530cc65943016df08b487f918097c749098eda52357274e0e2ccc05ae86"} Oct 30 00:05:28.165314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.165145 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c8c8757f-tspqd" event={"ID":"fd8f9a4a-5b77-4a52-9ea9-a12245c00489","Type":"ContainerDied","Data":"fc6457817a7ac391f70b9adcd5eccc2c79736f72ff0b2947e3fd0c9cb43efe1a"} Oct 30 00:05:28.165314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.165173 5001 scope.go:122] "RemoveContainer" containerID="1b8dd530cc65943016df08b487f918097c749098eda52357274e0e2ccc05ae86" Oct 30 00:05:28.174185 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.174168 5001 scope.go:122] "RemoveContainer" containerID="1b8dd530cc65943016df08b487f918097c749098eda52357274e0e2ccc05ae86" Oct 30 00:05:28.174506 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:05:28.174485 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b8dd530cc65943016df08b487f918097c749098eda52357274e0e2ccc05ae86\": container with ID starting with 1b8dd530cc65943016df08b487f918097c749098eda52357274e0e2ccc05ae86 not found: ID does not exist" containerID="1b8dd530cc65943016df08b487f918097c749098eda52357274e0e2ccc05ae86" Oct 30 00:05:28.174561 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.174515 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b8dd530cc65943016df08b487f918097c749098eda52357274e0e2ccc05ae86"} err="failed to get container status \"1b8dd530cc65943016df08b487f918097c749098eda52357274e0e2ccc05ae86\": rpc error: code = NotFound desc = could not find container \"1b8dd530cc65943016df08b487f918097c749098eda52357274e0e2ccc05ae86\": container with ID starting with 1b8dd530cc65943016df08b487f918097c749098eda52357274e0e2ccc05ae86 not found: ID does not exist" Oct 30 00:05:28.235738 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.235710 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-bound-sa-token\") pod \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " Oct 30 00:05:28.235892 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.235764 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-ca-trust-extracted\") pod \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " Oct 30 00:05:28.235892 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.235794 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-trusted-ca\") pod \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " Oct 30 00:05:28.236041 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.235902 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-tls\" (UniqueName: \"kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-tls\") pod \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " Oct 30 00:05:28.236041 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.235979 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-installation-pull-secrets\") pod \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " Oct 30 00:05:28.236041 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.236013 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-kube-api-access-f87wl\" (UniqueName: \"kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-kube-api-access-f87wl\") pod \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " Oct 30 00:05:28.236194 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.236047 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-certificates\") pod \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " Oct 30 00:05:28.236194 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.236087 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-image-registry-private-configuration\") pod \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\" (UID: \"fd8f9a4a-5b77-4a52-9ea9-a12245c00489\") " Oct 30 00:05:28.236312 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.236216 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-trusted-ca" pod "fd8f9a4a-5b77-4a52-9ea9-a12245c00489" (UID: "fd8f9a4a-5b77-4a52-9ea9-a12245c00489"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:05:28.236497 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.236461 5001 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-trusted-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:05:28.236497 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.236485 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-certificates" pod "fd8f9a4a-5b77-4a52-9ea9-a12245c00489" (UID: "fd8f9a4a-5b77-4a52-9ea9-a12245c00489"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:05:28.238739 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.238703 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-tls" pod "fd8f9a4a-5b77-4a52-9ea9-a12245c00489" (UID: "fd8f9a4a-5b77-4a52-9ea9-a12245c00489"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:05:28.238900 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.238871 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-kube-api-access-f87wl" pod "fd8f9a4a-5b77-4a52-9ea9-a12245c00489" (UID: "fd8f9a4a-5b77-4a52-9ea9-a12245c00489"). InnerVolumeSpecName "kube-api-access-f87wl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:05:28.239374 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.239351 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-image-registry-private-configuration" pod "fd8f9a4a-5b77-4a52-9ea9-a12245c00489" (UID: "fd8f9a4a-5b77-4a52-9ea9-a12245c00489"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:05:28.239616 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.239594 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-installation-pull-secrets" pod "fd8f9a4a-5b77-4a52-9ea9-a12245c00489" (UID: "fd8f9a4a-5b77-4a52-9ea9-a12245c00489"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:05:28.239834 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.239818 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-bound-sa-token" pod "fd8f9a4a-5b77-4a52-9ea9-a12245c00489" (UID: "fd8f9a4a-5b77-4a52-9ea9-a12245c00489"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:05:28.244666 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.244639 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-ca-trust-extracted" pod "fd8f9a4a-5b77-4a52-9ea9-a12245c00489" (UID: "fd8f9a4a-5b77-4a52-9ea9-a12245c00489"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:05:28.337306 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.337274 5001 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-image-registry-private-configuration\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:05:28.337306 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.337303 5001 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-bound-sa-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:05:28.337498 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.337315 5001 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-ca-trust-extracted\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:05:28.337498 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.337325 5001 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-tls\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:05:28.337498 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.337334 5001 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-installation-pull-secrets\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:05:28.337498 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.337343 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f87wl\" (UniqueName: \"kubernetes.io/projected/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-kube-api-access-f87wl\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:05:28.337498 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.337351 5001 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd8f9a4a-5b77-4a52-9ea9-a12245c00489-registry-certificates\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:05:28.511707 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.511672 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c8c8757f-tspqd"] Oct 30 00:05:28.521629 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.521594 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7c8c8757f-tspqd"] Oct 30 00:05:28.915666 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:28.915632 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd8f9a4a-5b77-4a52-9ea9-a12245c00489" path="/var/lib/kubelet/pods/fd8f9a4a-5b77-4a52-9ea9-a12245c00489/volumes" Oct 30 00:05:47.320900 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:05:47.320860 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:05:49.809125 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:49.809086 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" podUID="526cc561-1bbd-4079-8a2f-326f939753e0" containerName="registry" containerID="cri-o://aeaca81a572026cf1ae8ce575a28ae77ced4f7cc01bdee44413a58fb64cb0530" gracePeriod=30 Oct 30 00:05:50.058758 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.058728 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:05:50.108168 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.108090 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/526cc561-1bbd-4079-8a2f-326f939753e0-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/526cc561-1bbd-4079-8a2f-326f939753e0-image-registry-private-configuration\") pod \"526cc561-1bbd-4079-8a2f-326f939753e0\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " Oct 30 00:05:50.108168 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.108139 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-kube-api-access-cngbs\" (UniqueName: \"kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-kube-api-access-cngbs\") pod \"526cc561-1bbd-4079-8a2f-326f939753e0\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " Oct 30 00:05:50.108168 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.108166 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-bound-sa-token\") pod \"526cc561-1bbd-4079-8a2f-326f939753e0\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " Oct 30 00:05:50.108423 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.108208 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/526cc561-1bbd-4079-8a2f-326f939753e0-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/526cc561-1bbd-4079-8a2f-326f939753e0-trusted-ca\") pod \"526cc561-1bbd-4079-8a2f-326f939753e0\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " Oct 30 00:05:50.108423 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.108229 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/526cc561-1bbd-4079-8a2f-326f939753e0-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/526cc561-1bbd-4079-8a2f-326f939753e0-ca-trust-extracted\") pod \"526cc561-1bbd-4079-8a2f-326f939753e0\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " Oct 30 00:05:50.108423 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.108249 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/526cc561-1bbd-4079-8a2f-326f939753e0-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/526cc561-1bbd-4079-8a2f-326f939753e0-installation-pull-secrets\") pod \"526cc561-1bbd-4079-8a2f-326f939753e0\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " Oct 30 00:05:50.108423 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.108284 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/526cc561-1bbd-4079-8a2f-326f939753e0-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/526cc561-1bbd-4079-8a2f-326f939753e0-registry-certificates\") pod \"526cc561-1bbd-4079-8a2f-326f939753e0\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " Oct 30 00:05:50.108423 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.108325 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-registry-tls\" (UniqueName: \"kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-registry-tls\") pod \"526cc561-1bbd-4079-8a2f-326f939753e0\" (UID: \"526cc561-1bbd-4079-8a2f-326f939753e0\") " Oct 30 00:05:50.108800 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.108747 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/526cc561-1bbd-4079-8a2f-326f939753e0-trusted-ca" pod "526cc561-1bbd-4079-8a2f-326f939753e0" (UID: "526cc561-1bbd-4079-8a2f-326f939753e0"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:05:50.108956 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.108907 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/526cc561-1bbd-4079-8a2f-326f939753e0-registry-certificates" pod "526cc561-1bbd-4079-8a2f-326f939753e0" (UID: "526cc561-1bbd-4079-8a2f-326f939753e0"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:05:50.110954 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.110921 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/526cc561-1bbd-4079-8a2f-326f939753e0-image-registry-private-configuration" pod "526cc561-1bbd-4079-8a2f-326f939753e0" (UID: "526cc561-1bbd-4079-8a2f-326f939753e0"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:05:50.111073 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.110997 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-bound-sa-token" pod "526cc561-1bbd-4079-8a2f-326f939753e0" (UID: "526cc561-1bbd-4079-8a2f-326f939753e0"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:05:50.111073 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.111035 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-registry-tls" pod "526cc561-1bbd-4079-8a2f-326f939753e0" (UID: "526cc561-1bbd-4079-8a2f-326f939753e0"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:05:50.111193 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.111087 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-kube-api-access-cngbs" pod "526cc561-1bbd-4079-8a2f-326f939753e0" (UID: "526cc561-1bbd-4079-8a2f-326f939753e0"). InnerVolumeSpecName "kube-api-access-cngbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:05:50.112148 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.112119 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/526cc561-1bbd-4079-8a2f-326f939753e0-installation-pull-secrets" pod "526cc561-1bbd-4079-8a2f-326f939753e0" (UID: "526cc561-1bbd-4079-8a2f-326f939753e0"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:05:50.115695 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.115669 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/526cc561-1bbd-4079-8a2f-326f939753e0-ca-trust-extracted" pod "526cc561-1bbd-4079-8a2f-326f939753e0" (UID: "526cc561-1bbd-4079-8a2f-326f939753e0"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:05:50.209581 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.209544 5001 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/526cc561-1bbd-4079-8a2f-326f939753e0-trusted-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:05:50.209581 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.209583 5001 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/526cc561-1bbd-4079-8a2f-326f939753e0-ca-trust-extracted\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:05:50.209778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.209597 5001 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/526cc561-1bbd-4079-8a2f-326f939753e0-installation-pull-secrets\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:05:50.209778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.209606 5001 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/526cc561-1bbd-4079-8a2f-326f939753e0-registry-certificates\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:05:50.209778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.209616 5001 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-registry-tls\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:05:50.209778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.209625 5001 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/526cc561-1bbd-4079-8a2f-326f939753e0-image-registry-private-configuration\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:05:50.209778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.209635 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cngbs\" (UniqueName: \"kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-kube-api-access-cngbs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:05:50.209778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.209644 5001 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/526cc561-1bbd-4079-8a2f-326f939753e0-bound-sa-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:05:50.250471 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.250439 5001 generic.go:356] "Generic (PLEG): container finished" podID="526cc561-1bbd-4079-8a2f-326f939753e0" containerID="aeaca81a572026cf1ae8ce575a28ae77ced4f7cc01bdee44413a58fb64cb0530" exitCode=0 Oct 30 00:05:50.250639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.250484 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" Oct 30 00:05:50.250639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.250507 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" event={"ID":"526cc561-1bbd-4079-8a2f-326f939753e0","Type":"ContainerDied","Data":"aeaca81a572026cf1ae8ce575a28ae77ced4f7cc01bdee44413a58fb64cb0530"} Oct 30 00:05:50.250639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.250558 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7dcddbb874-9zx5v" event={"ID":"526cc561-1bbd-4079-8a2f-326f939753e0","Type":"ContainerDied","Data":"06d9d93ac69d98dd8430cd691fecfc93a0fed308d287385ee18201c59866cd3a"} Oct 30 00:05:50.250639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.250593 5001 scope.go:122] "RemoveContainer" containerID="aeaca81a572026cf1ae8ce575a28ae77ced4f7cc01bdee44413a58fb64cb0530" Oct 30 00:05:50.259502 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.259484 5001 scope.go:122] "RemoveContainer" containerID="aeaca81a572026cf1ae8ce575a28ae77ced4f7cc01bdee44413a58fb64cb0530" Oct 30 00:05:50.259756 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:05:50.259735 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeaca81a572026cf1ae8ce575a28ae77ced4f7cc01bdee44413a58fb64cb0530\": container with ID starting with aeaca81a572026cf1ae8ce575a28ae77ced4f7cc01bdee44413a58fb64cb0530 not found: ID does not exist" containerID="aeaca81a572026cf1ae8ce575a28ae77ced4f7cc01bdee44413a58fb64cb0530" Oct 30 00:05:50.259824 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.259764 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeaca81a572026cf1ae8ce575a28ae77ced4f7cc01bdee44413a58fb64cb0530"} err="failed to get container status \"aeaca81a572026cf1ae8ce575a28ae77ced4f7cc01bdee44413a58fb64cb0530\": rpc error: code = NotFound desc = could not find container \"aeaca81a572026cf1ae8ce575a28ae77ced4f7cc01bdee44413a58fb64cb0530\": container with ID starting with aeaca81a572026cf1ae8ce575a28ae77ced4f7cc01bdee44413a58fb64cb0530 not found: ID does not exist" Oct 30 00:05:50.294419 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.294396 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7dcddbb874-9zx5v"] Oct 30 00:05:50.297736 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.297715 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7dcddbb874-9zx5v"] Oct 30 00:05:50.919246 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:50.919211 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="526cc561-1bbd-4079-8a2f-326f939753e0" path="/var/lib/kubelet/pods/526cc561-1bbd-4079-8a2f-326f939753e0/volumes" Oct 30 00:05:55.272661 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:55.272630 5001 generic.go:356] "Generic (PLEG): container finished" podID="c1a647f1-e715-4a8a-8b44-0196b3b1af27" containerID="a4959dc8121b76faa5b758fa2cfcb5a2373e16e833f2613254ce12fad0ed987c" exitCode=0 Oct 30 00:05:55.273303 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:55.272696 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l" event={"ID":"c1a647f1-e715-4a8a-8b44-0196b3b1af27","Type":"ContainerDied","Data":"a4959dc8121b76faa5b758fa2cfcb5a2373e16e833f2613254ce12fad0ed987c"} Oct 30 00:05:55.273303 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:55.272999 5001 scope.go:122] "RemoveContainer" containerID="a4959dc8121b76faa5b758fa2cfcb5a2373e16e833f2613254ce12fad0ed987c" Oct 30 00:05:56.278412 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:05:56.278374 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfz9l" event={"ID":"c1a647f1-e715-4a8a-8b44-0196b3b1af27","Type":"ContainerStarted","Data":"27706f7a9f0dc16196786b928141623d69b1dc4cd1b7c6adac392b17a7603afe"} Oct 30 00:06:15.353993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:15.353959 5001 generic.go:356] "Generic (PLEG): container finished" podID="33f61ea8-5b1a-405a-aa36-7f07d79e2ec8" containerID="06d35bd282ecc75a88f7686df991b0bfa14b3b765a5cf4ad69487217eed30751" exitCode=0 Oct 30 00:06:15.354396 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:15.354041 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-nzxzl" event={"ID":"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8","Type":"ContainerDied","Data":"06d35bd282ecc75a88f7686df991b0bfa14b3b765a5cf4ad69487217eed30751"} Oct 30 00:06:15.354396 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:15.354347 5001 scope.go:122] "RemoveContainer" containerID="06d35bd282ecc75a88f7686df991b0bfa14b3b765a5cf4ad69487217eed30751" Oct 30 00:06:15.876279 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:15.876245 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-nkc49-p6m54"] Oct 30 00:06:15.876508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:15.876473 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-nkc49-p6m54" podUID="09f5d7e2-51b8-4153-89c5-674d67065762" containerName="insights-gathering" containerID="cri-o://56f23de064fa0cb6e1bd391a754c29fac230383af672fc16c364bcc6cab65f7d" gracePeriod=30 Oct 30 00:06:16.060060 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.060031 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-f4dn8-58cbq"] Oct 30 00:06:16.060330 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.060318 5001 cpu_manager.go:414] "containerMap: removing container" podUID="fd8f9a4a-5b77-4a52-9ea9-a12245c00489" containerName="registry" Oct 30 00:06:16.060373 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.060331 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd8f9a4a-5b77-4a52-9ea9-a12245c00489" containerName="registry" Oct 30 00:06:16.060373 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.060343 5001 cpu_manager.go:414] "containerMap: removing container" podUID="526cc561-1bbd-4079-8a2f-326f939753e0" containerName="registry" Oct 30 00:06:16.060373 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.060348 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="526cc561-1bbd-4079-8a2f-326f939753e0" containerName="registry" Oct 30 00:06:16.060519 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.060402 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="526cc561-1bbd-4079-8a2f-326f939753e0" containerName="registry" Oct 30 00:06:16.060519 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.060416 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd8f9a4a-5b77-4a52-9ea9-a12245c00489" containerName="registry" Oct 30 00:06:16.063435 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.063410 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f4dn8-58cbq" Oct 30 00:06:16.084975 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.084885 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-f4dn8-58cbq"] Oct 30 00:06:16.165288 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.165265 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-nkc49-p6m54" Oct 30 00:06:16.208355 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.208319 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1a85edc-b54d-4c50-9012-87f4d06ab934-service-ca-bundle\") pod \"periodic-gathering-f4dn8-58cbq\" (UID: \"e1a85edc-b54d-4c50-9012-87f4d06ab934\") " pod="openshift-insights/periodic-gathering-f4dn8-58cbq" Oct 30 00:06:16.208517 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.208402 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e1a85edc-b54d-4c50-9012-87f4d06ab934-archives-path\") pod \"periodic-gathering-f4dn8-58cbq\" (UID: \"e1a85edc-b54d-4c50-9012-87f4d06ab934\") " pod="openshift-insights/periodic-gathering-f4dn8-58cbq" Oct 30 00:06:16.208517 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.208435 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99ssg\" (UniqueName: \"kubernetes.io/projected/e1a85edc-b54d-4c50-9012-87f4d06ab934-kube-api-access-99ssg\") pod \"periodic-gathering-f4dn8-58cbq\" (UID: \"e1a85edc-b54d-4c50-9012-87f4d06ab934\") " pod="openshift-insights/periodic-gathering-f4dn8-58cbq" Oct 30 00:06:16.208517 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.208508 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1a85edc-b54d-4c50-9012-87f4d06ab934-trusted-ca-bundle\") pod \"periodic-gathering-f4dn8-58cbq\" (UID: \"e1a85edc-b54d-4c50-9012-87f4d06ab934\") " pod="openshift-insights/periodic-gathering-f4dn8-58cbq" Oct 30 00:06:16.309140 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.309104 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f5d7e2-51b8-4153-89c5-674d67065762-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/09f5d7e2-51b8-4153-89c5-674d67065762-archives-path\") pod \"09f5d7e2-51b8-4153-89c5-674d67065762\" (UID: \"09f5d7e2-51b8-4153-89c5-674d67065762\") " Oct 30 00:06:16.309334 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.309161 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09f5d7e2-51b8-4153-89c5-674d67065762-kube-api-access-lgkdb\" (UniqueName: \"kubernetes.io/projected/09f5d7e2-51b8-4153-89c5-674d67065762-kube-api-access-lgkdb\") pod \"09f5d7e2-51b8-4153-89c5-674d67065762\" (UID: \"09f5d7e2-51b8-4153-89c5-674d67065762\") " Oct 30 00:06:16.309334 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.309193 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09f5d7e2-51b8-4153-89c5-674d67065762-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09f5d7e2-51b8-4153-89c5-674d67065762-trusted-ca-bundle\") pod \"09f5d7e2-51b8-4153-89c5-674d67065762\" (UID: \"09f5d7e2-51b8-4153-89c5-674d67065762\") " Oct 30 00:06:16.309449 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.309331 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09f5d7e2-51b8-4153-89c5-674d67065762-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09f5d7e2-51b8-4153-89c5-674d67065762-service-ca-bundle\") pod \"09f5d7e2-51b8-4153-89c5-674d67065762\" (UID: \"09f5d7e2-51b8-4153-89c5-674d67065762\") " Oct 30 00:06:16.309449 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.309431 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f5d7e2-51b8-4153-89c5-674d67065762-archives-path" pod "09f5d7e2-51b8-4153-89c5-674d67065762" (UID: "09f5d7e2-51b8-4153-89c5-674d67065762"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:06:16.309550 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.309473 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1a85edc-b54d-4c50-9012-87f4d06ab934-service-ca-bundle\") pod \"periodic-gathering-f4dn8-58cbq\" (UID: \"e1a85edc-b54d-4c50-9012-87f4d06ab934\") " pod="openshift-insights/periodic-gathering-f4dn8-58cbq" Oct 30 00:06:16.309550 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.309529 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e1a85edc-b54d-4c50-9012-87f4d06ab934-archives-path\") pod \"periodic-gathering-f4dn8-58cbq\" (UID: \"e1a85edc-b54d-4c50-9012-87f4d06ab934\") " pod="openshift-insights/periodic-gathering-f4dn8-58cbq" Oct 30 00:06:16.309650 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.309542 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09f5d7e2-51b8-4153-89c5-674d67065762-trusted-ca-bundle" pod "09f5d7e2-51b8-4153-89c5-674d67065762" (UID: "09f5d7e2-51b8-4153-89c5-674d67065762"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:06:16.309650 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.309561 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99ssg\" (UniqueName: \"kubernetes.io/projected/e1a85edc-b54d-4c50-9012-87f4d06ab934-kube-api-access-99ssg\") pod \"periodic-gathering-f4dn8-58cbq\" (UID: \"e1a85edc-b54d-4c50-9012-87f4d06ab934\") " pod="openshift-insights/periodic-gathering-f4dn8-58cbq" Oct 30 00:06:16.309650 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.309625 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1a85edc-b54d-4c50-9012-87f4d06ab934-trusted-ca-bundle\") pod \"periodic-gathering-f4dn8-58cbq\" (UID: \"e1a85edc-b54d-4c50-9012-87f4d06ab934\") " pod="openshift-insights/periodic-gathering-f4dn8-58cbq" Oct 30 00:06:16.309785 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.309652 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09f5d7e2-51b8-4153-89c5-674d67065762-service-ca-bundle" pod "09f5d7e2-51b8-4153-89c5-674d67065762" (UID: "09f5d7e2-51b8-4153-89c5-674d67065762"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:06:16.309785 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.309704 5001 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/09f5d7e2-51b8-4153-89c5-674d67065762-archives-path\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:06:16.309785 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.309720 5001 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09f5d7e2-51b8-4153-89c5-674d67065762-trusted-ca-bundle\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:06:16.309885 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.309843 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e1a85edc-b54d-4c50-9012-87f4d06ab934-archives-path\") pod \"periodic-gathering-f4dn8-58cbq\" (UID: \"e1a85edc-b54d-4c50-9012-87f4d06ab934\") " pod="openshift-insights/periodic-gathering-f4dn8-58cbq" Oct 30 00:06:16.310249 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.310227 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1a85edc-b54d-4c50-9012-87f4d06ab934-service-ca-bundle\") pod \"periodic-gathering-f4dn8-58cbq\" (UID: \"e1a85edc-b54d-4c50-9012-87f4d06ab934\") " pod="openshift-insights/periodic-gathering-f4dn8-58cbq" Oct 30 00:06:16.310585 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.310565 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1a85edc-b54d-4c50-9012-87f4d06ab934-trusted-ca-bundle\") pod \"periodic-gathering-f4dn8-58cbq\" (UID: \"e1a85edc-b54d-4c50-9012-87f4d06ab934\") " pod="openshift-insights/periodic-gathering-f4dn8-58cbq" Oct 30 00:06:16.311559 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.311541 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f5d7e2-51b8-4153-89c5-674d67065762-kube-api-access-lgkdb" pod "09f5d7e2-51b8-4153-89c5-674d67065762" (UID: "09f5d7e2-51b8-4153-89c5-674d67065762"). InnerVolumeSpecName "kube-api-access-lgkdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:06:16.330844 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.330824 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99ssg\" (UniqueName: \"kubernetes.io/projected/e1a85edc-b54d-4c50-9012-87f4d06ab934-kube-api-access-99ssg\") pod \"periodic-gathering-f4dn8-58cbq\" (UID: \"e1a85edc-b54d-4c50-9012-87f4d06ab934\") " pod="openshift-insights/periodic-gathering-f4dn8-58cbq" Oct 30 00:06:16.358748 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.358724 5001 generic.go:356] "Generic (PLEG): container finished" podID="8a82abd0-5057-4361-9b0d-7a4f23ae97d1" containerID="4cbd23ae981f19cba837871a1c38bcf515259628b6992cfd909fbeb1b3742a4a" exitCode=0 Oct 30 00:06:16.359101 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.358792 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6d24p" event={"ID":"8a82abd0-5057-4361-9b0d-7a4f23ae97d1","Type":"ContainerDied","Data":"4cbd23ae981f19cba837871a1c38bcf515259628b6992cfd909fbeb1b3742a4a"} Oct 30 00:06:16.359152 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.359116 5001 scope.go:122] "RemoveContainer" containerID="4cbd23ae981f19cba837871a1c38bcf515259628b6992cfd909fbeb1b3742a4a" Oct 30 00:06:16.360008 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.359988 5001 generic.go:356] "Generic (PLEG): container finished" podID="09f5d7e2-51b8-4153-89c5-674d67065762" containerID="56f23de064fa0cb6e1bd391a754c29fac230383af672fc16c364bcc6cab65f7d" exitCode=2 Oct 30 00:06:16.360108 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.360050 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-nkc49-p6m54" Oct 30 00:06:16.360108 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.360049 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-nkc49-p6m54" event={"ID":"09f5d7e2-51b8-4153-89c5-674d67065762","Type":"ContainerDied","Data":"56f23de064fa0cb6e1bd391a754c29fac230383af672fc16c364bcc6cab65f7d"} Oct 30 00:06:16.360227 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.360112 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-nkc49-p6m54" event={"ID":"09f5d7e2-51b8-4153-89c5-674d67065762","Type":"ContainerDied","Data":"115ab9e9fecc7cf88524a0ecdc1649c7caec7db2b46b6428070d63a5fbf22474"} Oct 30 00:06:16.360227 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.360132 5001 scope.go:122] "RemoveContainer" containerID="56f23de064fa0cb6e1bd391a754c29fac230383af672fc16c364bcc6cab65f7d" Oct 30 00:06:16.362070 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.362021 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-nzxzl" event={"ID":"33f61ea8-5b1a-405a-aa36-7f07d79e2ec8","Type":"ContainerStarted","Data":"028c5ea1e2cfbcd02126072153be57b57d06812c470d28d4bf8f0912141fa510"} Oct 30 00:06:16.378988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.378965 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f4dn8-58cbq" Oct 30 00:06:16.380513 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.380493 5001 scope.go:122] "RemoveContainer" containerID="56f23de064fa0cb6e1bd391a754c29fac230383af672fc16c364bcc6cab65f7d" Oct 30 00:06:16.380856 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:06:16.380818 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56f23de064fa0cb6e1bd391a754c29fac230383af672fc16c364bcc6cab65f7d\": container with ID starting with 56f23de064fa0cb6e1bd391a754c29fac230383af672fc16c364bcc6cab65f7d not found: ID does not exist" containerID="56f23de064fa0cb6e1bd391a754c29fac230383af672fc16c364bcc6cab65f7d" Oct 30 00:06:16.380967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.380855 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56f23de064fa0cb6e1bd391a754c29fac230383af672fc16c364bcc6cab65f7d"} err="failed to get container status \"56f23de064fa0cb6e1bd391a754c29fac230383af672fc16c364bcc6cab65f7d\": rpc error: code = NotFound desc = could not find container \"56f23de064fa0cb6e1bd391a754c29fac230383af672fc16c364bcc6cab65f7d\": container with ID starting with 56f23de064fa0cb6e1bd391a754c29fac230383af672fc16c364bcc6cab65f7d not found: ID does not exist" Oct 30 00:06:16.410760 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.410691 5001 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09f5d7e2-51b8-4153-89c5-674d67065762-service-ca-bundle\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:06:16.410760 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.410716 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgkdb\" (UniqueName: \"kubernetes.io/projected/09f5d7e2-51b8-4153-89c5-674d67065762-kube-api-access-lgkdb\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:06:16.476049 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.476014 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-nkc49-p6m54"] Oct 30 00:06:16.480257 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.480230 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-nkc49-p6m54"] Oct 30 00:06:16.678363 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.678283 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-f4dn8-58cbq"] Oct 30 00:06:16.683328 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:06:16.683298 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1a85edc_b54d_4c50_9012_87f4d06ab934.slice/crio-58f6280e4da46baa53f2920cd8ea9cb1dbb91bc8c190b46408dfb918bb7ef0e6 WatchSource:0}: Error finding container 58f6280e4da46baa53f2920cd8ea9cb1dbb91bc8c190b46408dfb918bb7ef0e6: Status 404 returned error can't find the container with id 58f6280e4da46baa53f2920cd8ea9cb1dbb91bc8c190b46408dfb918bb7ef0e6 Oct 30 00:06:16.915320 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:16.915290 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09f5d7e2-51b8-4153-89c5-674d67065762" path="/var/lib/kubelet/pods/09f5d7e2-51b8-4153-89c5-674d67065762/volumes" Oct 30 00:06:17.320658 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:06:17.320624 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:06:17.368580 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:17.368544 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f4dn8-58cbq" event={"ID":"e1a85edc-b54d-4c50-9012-87f4d06ab934","Type":"ContainerStarted","Data":"b88e285078d466944daafeda435abf21bc402690bf009495533e5735cbc242fe"} Oct 30 00:06:17.369046 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:17.368596 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f4dn8-58cbq" event={"ID":"e1a85edc-b54d-4c50-9012-87f4d06ab934","Type":"ContainerStarted","Data":"58f6280e4da46baa53f2920cd8ea9cb1dbb91bc8c190b46408dfb918bb7ef0e6"} Oct 30 00:06:17.370102 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:17.370078 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6d24p" event={"ID":"8a82abd0-5057-4361-9b0d-7a4f23ae97d1","Type":"ContainerStarted","Data":"786f468f7386fb04c9eba2cb9596fd32dc8911d086afab313de32bcac52cbc73"} Oct 30 00:06:17.404211 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:06:17.404168 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-f4dn8-58cbq" podStartSLOduration=1.404155679 podStartE2EDuration="1.404155679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:06:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:06:17.403956364 +0000 UTC m=+92.609457546" watchObservedRunningTime="2026-10-30 00:06:17.404155679 +0000 UTC m=+92.609656857" Oct 30 00:06:47.321360 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:06:47.321316 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:07:17.321354 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:07:17.321272 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:07:40.676821 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:07:40.676740 5001 generic.go:356] "Generic (PLEG): container finished" podID="e1a85edc-b54d-4c50-9012-87f4d06ab934" containerID="b88e285078d466944daafeda435abf21bc402690bf009495533e5735cbc242fe" exitCode=0 Oct 30 00:07:40.676821 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:07:40.676792 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f4dn8-58cbq" event={"ID":"e1a85edc-b54d-4c50-9012-87f4d06ab934","Type":"ContainerDied","Data":"b88e285078d466944daafeda435abf21bc402690bf009495533e5735cbc242fe"} Oct 30 00:07:41.802973 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:07:41.802947 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f4dn8-58cbq" Oct 30 00:07:41.849121 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:07:41.849094 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1a85edc-b54d-4c50-9012-87f4d06ab934-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1a85edc-b54d-4c50-9012-87f4d06ab934-service-ca-bundle\") pod \"e1a85edc-b54d-4c50-9012-87f4d06ab934\" (UID: \"e1a85edc-b54d-4c50-9012-87f4d06ab934\") " Oct 30 00:07:41.849277 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:07:41.849144 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1a85edc-b54d-4c50-9012-87f4d06ab934-kube-api-access-99ssg\" (UniqueName: \"kubernetes.io/projected/e1a85edc-b54d-4c50-9012-87f4d06ab934-kube-api-access-99ssg\") pod \"e1a85edc-b54d-4c50-9012-87f4d06ab934\" (UID: \"e1a85edc-b54d-4c50-9012-87f4d06ab934\") " Oct 30 00:07:41.849277 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:07:41.849182 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1a85edc-b54d-4c50-9012-87f4d06ab934-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1a85edc-b54d-4c50-9012-87f4d06ab934-trusted-ca-bundle\") pod \"e1a85edc-b54d-4c50-9012-87f4d06ab934\" (UID: \"e1a85edc-b54d-4c50-9012-87f4d06ab934\") " Oct 30 00:07:41.849277 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:07:41.849208 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1a85edc-b54d-4c50-9012-87f4d06ab934-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e1a85edc-b54d-4c50-9012-87f4d06ab934-archives-path\") pod \"e1a85edc-b54d-4c50-9012-87f4d06ab934\" (UID: \"e1a85edc-b54d-4c50-9012-87f4d06ab934\") " Oct 30 00:07:41.849549 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:07:41.849514 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1a85edc-b54d-4c50-9012-87f4d06ab934-service-ca-bundle" pod "e1a85edc-b54d-4c50-9012-87f4d06ab934" (UID: "e1a85edc-b54d-4c50-9012-87f4d06ab934"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:07:41.849668 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:07:41.849610 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1a85edc-b54d-4c50-9012-87f4d06ab934-trusted-ca-bundle" pod "e1a85edc-b54d-4c50-9012-87f4d06ab934" (UID: "e1a85edc-b54d-4c50-9012-87f4d06ab934"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:07:41.849668 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:07:41.849610 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1a85edc-b54d-4c50-9012-87f4d06ab934-archives-path" pod "e1a85edc-b54d-4c50-9012-87f4d06ab934" (UID: "e1a85edc-b54d-4c50-9012-87f4d06ab934"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:07:41.851330 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:07:41.851300 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1a85edc-b54d-4c50-9012-87f4d06ab934-kube-api-access-99ssg" pod "e1a85edc-b54d-4c50-9012-87f4d06ab934" (UID: "e1a85edc-b54d-4c50-9012-87f4d06ab934"). InnerVolumeSpecName "kube-api-access-99ssg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:07:41.950562 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:07:41.950480 5001 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1a85edc-b54d-4c50-9012-87f4d06ab934-service-ca-bundle\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:07:41.950562 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:07:41.950511 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99ssg\" (UniqueName: \"kubernetes.io/projected/e1a85edc-b54d-4c50-9012-87f4d06ab934-kube-api-access-99ssg\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:07:41.950562 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:07:41.950522 5001 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1a85edc-b54d-4c50-9012-87f4d06ab934-trusted-ca-bundle\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:07:41.950562 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:07:41.950532 5001 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e1a85edc-b54d-4c50-9012-87f4d06ab934-archives-path\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:07:42.686037 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:07:42.685997 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f4dn8-58cbq" event={"ID":"e1a85edc-b54d-4c50-9012-87f4d06ab934","Type":"ContainerDied","Data":"58f6280e4da46baa53f2920cd8ea9cb1dbb91bc8c190b46408dfb918bb7ef0e6"} Oct 30 00:07:42.686037 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:07:42.686021 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f4dn8-58cbq" Oct 30 00:07:42.686319 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:07:42.686050 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58f6280e4da46baa53f2920cd8ea9cb1dbb91bc8c190b46408dfb918bb7ef0e6" Oct 30 00:07:47.320728 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:07:47.320693 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:08:17.321880 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:08:17.321845 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:08:28.833235 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:28.833162 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716pmws"] Oct 30 00:08:28.833610 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:28.833485 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e1a85edc-b54d-4c50-9012-87f4d06ab934" containerName="insights-gathering" Oct 30 00:08:28.833610 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:28.833498 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1a85edc-b54d-4c50-9012-87f4d06ab934" containerName="insights-gathering" Oct 30 00:08:28.833610 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:28.833524 5001 cpu_manager.go:414] "containerMap: removing container" podUID="09f5d7e2-51b8-4153-89c5-674d67065762" containerName="insights-gathering" Oct 30 00:08:28.833610 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:28.833530 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09f5d7e2-51b8-4153-89c5-674d67065762" containerName="insights-gathering" Oct 30 00:08:28.833610 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:28.833586 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="09f5d7e2-51b8-4153-89c5-674d67065762" containerName="insights-gathering" Oct 30 00:08:28.833610 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:28.833599 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1a85edc-b54d-4c50-9012-87f4d06ab934" containerName="insights-gathering" Oct 30 00:08:28.836667 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:28.836647 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716pmws" Oct 30 00:08:28.862202 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:28.862175 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 00:08:28.862348 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:28.862192 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-4tx68\"" Oct 30 00:08:28.862348 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:28.862293 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 00:08:28.863278 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:28.863252 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716pmws"] Oct 30 00:08:28.994355 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:28.994317 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4484ecc5-88d4-4f8e-8244-a7b35f95bb3a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716pmws\" (UID: \"4484ecc5-88d4-4f8e-8244-a7b35f95bb3a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716pmws" Oct 30 00:08:28.994525 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:28.994427 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldnhb\" (UniqueName: \"kubernetes.io/projected/4484ecc5-88d4-4f8e-8244-a7b35f95bb3a-kube-api-access-ldnhb\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716pmws\" (UID: \"4484ecc5-88d4-4f8e-8244-a7b35f95bb3a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716pmws" Oct 30 00:08:28.994525 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:28.994476 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4484ecc5-88d4-4f8e-8244-a7b35f95bb3a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716pmws\" (UID: \"4484ecc5-88d4-4f8e-8244-a7b35f95bb3a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716pmws" Oct 30 00:08:29.095689 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:29.095600 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4484ecc5-88d4-4f8e-8244-a7b35f95bb3a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716pmws\" (UID: \"4484ecc5-88d4-4f8e-8244-a7b35f95bb3a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716pmws" Oct 30 00:08:29.095689 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:29.095665 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4484ecc5-88d4-4f8e-8244-a7b35f95bb3a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716pmws\" (UID: \"4484ecc5-88d4-4f8e-8244-a7b35f95bb3a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716pmws" Oct 30 00:08:29.095934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:29.095695 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldnhb\" (UniqueName: \"kubernetes.io/projected/4484ecc5-88d4-4f8e-8244-a7b35f95bb3a-kube-api-access-ldnhb\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716pmws\" (UID: \"4484ecc5-88d4-4f8e-8244-a7b35f95bb3a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716pmws" Oct 30 00:08:29.096038 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:29.096016 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4484ecc5-88d4-4f8e-8244-a7b35f95bb3a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716pmws\" (UID: \"4484ecc5-88d4-4f8e-8244-a7b35f95bb3a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716pmws" Oct 30 00:08:29.096108 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:29.096085 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4484ecc5-88d4-4f8e-8244-a7b35f95bb3a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716pmws\" (UID: \"4484ecc5-88d4-4f8e-8244-a7b35f95bb3a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716pmws" Oct 30 00:08:29.117287 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:29.117250 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldnhb\" (UniqueName: \"kubernetes.io/projected/4484ecc5-88d4-4f8e-8244-a7b35f95bb3a-kube-api-access-ldnhb\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716pmws\" (UID: \"4484ecc5-88d4-4f8e-8244-a7b35f95bb3a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716pmws" Oct 30 00:08:29.152599 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:29.152572 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716pmws" Oct 30 00:08:29.444565 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:29.444492 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716pmws"] Oct 30 00:08:29.449117 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:08:29.449088 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4484ecc5_88d4_4f8e_8244_a7b35f95bb3a.slice/crio-d4e7eaeb08f3a895fd759fb62b18e9ad823e2044e2dba5c0a20da19afe7a4b96 WatchSource:0}: Error finding container d4e7eaeb08f3a895fd759fb62b18e9ad823e2044e2dba5c0a20da19afe7a4b96: Status 404 returned error can't find the container with id d4e7eaeb08f3a895fd759fb62b18e9ad823e2044e2dba5c0a20da19afe7a4b96 Oct 30 00:08:29.853813 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:29.853777 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716pmws" event={"ID":"4484ecc5-88d4-4f8e-8244-a7b35f95bb3a","Type":"ContainerStarted","Data":"d4e7eaeb08f3a895fd759fb62b18e9ad823e2044e2dba5c0a20da19afe7a4b96"} Oct 30 00:08:30.248631 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.248557 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c25w57"] Oct 30 00:08:30.252417 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.252390 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c25w57" Oct 30 00:08:30.274463 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.274440 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-cw9tt\"" Oct 30 00:08:30.276202 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.276181 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c25w57"] Oct 30 00:08:30.345576 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.345538 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c665gj4"] Oct 30 00:08:30.349527 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.349502 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c665gj4" Oct 30 00:08:30.368475 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.368108 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-879kz\"" Oct 30 00:08:30.368475 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.368408 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c665gj4"] Oct 30 00:08:30.405844 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.405813 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d34d892-f9ab-4bd2-920c-2c7305c7c57b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c25w57\" (UID: \"5d34d892-f9ab-4bd2-920c-2c7305c7c57b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c25w57" Oct 30 00:08:30.406085 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.405851 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d34d892-f9ab-4bd2-920c-2c7305c7c57b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c25w57\" (UID: \"5d34d892-f9ab-4bd2-920c-2c7305c7c57b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c25w57" Oct 30 00:08:30.406085 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.405877 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l768r\" (UniqueName: \"kubernetes.io/projected/5d34d892-f9ab-4bd2-920c-2c7305c7c57b-kube-api-access-l768r\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c25w57\" (UID: \"5d34d892-f9ab-4bd2-920c-2c7305c7c57b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c25w57" Oct 30 00:08:30.446830 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.446797 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x4tph"] Oct 30 00:08:30.450354 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.450333 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x4tph" Oct 30 00:08:30.462628 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.462601 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-kvdb4\"" Oct 30 00:08:30.469583 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.469518 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x4tph"] Oct 30 00:08:30.507295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.507216 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d34d892-f9ab-4bd2-920c-2c7305c7c57b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c25w57\" (UID: \"5d34d892-f9ab-4bd2-920c-2c7305c7c57b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c25w57" Oct 30 00:08:30.507295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.507252 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d34d892-f9ab-4bd2-920c-2c7305c7c57b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c25w57\" (UID: \"5d34d892-f9ab-4bd2-920c-2c7305c7c57b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c25w57" Oct 30 00:08:30.507295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.507278 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l768r\" (UniqueName: \"kubernetes.io/projected/5d34d892-f9ab-4bd2-920c-2c7305c7c57b-kube-api-access-l768r\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c25w57\" (UID: \"5d34d892-f9ab-4bd2-920c-2c7305c7c57b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c25w57" Oct 30 00:08:30.507561 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.507310 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d1bbe05-289d-414a-8854-db469519e37e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c665gj4\" (UID: \"7d1bbe05-289d-414a-8854-db469519e37e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c665gj4" Oct 30 00:08:30.507561 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.507356 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fkbq\" (UniqueName: \"kubernetes.io/projected/7d1bbe05-289d-414a-8854-db469519e37e-kube-api-access-7fkbq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c665gj4\" (UID: \"7d1bbe05-289d-414a-8854-db469519e37e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c665gj4" Oct 30 00:08:30.507561 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.507500 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d1bbe05-289d-414a-8854-db469519e37e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c665gj4\" (UID: \"7d1bbe05-289d-414a-8854-db469519e37e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c665gj4" Oct 30 00:08:30.507716 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.507664 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d34d892-f9ab-4bd2-920c-2c7305c7c57b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c25w57\" (UID: \"5d34d892-f9ab-4bd2-920c-2c7305c7c57b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c25w57" Oct 30 00:08:30.507716 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.507673 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d34d892-f9ab-4bd2-920c-2c7305c7c57b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c25w57\" (UID: \"5d34d892-f9ab-4bd2-920c-2c7305c7c57b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c25w57" Oct 30 00:08:30.528842 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.528811 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l768r\" (UniqueName: \"kubernetes.io/projected/5d34d892-f9ab-4bd2-920c-2c7305c7c57b-kube-api-access-l768r\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c25w57\" (UID: \"5d34d892-f9ab-4bd2-920c-2c7305c7c57b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c25w57" Oct 30 00:08:30.553775 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.553743 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5w4t2"] Oct 30 00:08:30.557786 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.557760 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5w4t2" Oct 30 00:08:30.570260 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.570229 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c25w57" Oct 30 00:08:30.570260 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.570247 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-6qtqd\"" Oct 30 00:08:30.577188 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.576996 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5w4t2"] Oct 30 00:08:30.608262 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.608231 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fkbq\" (UniqueName: \"kubernetes.io/projected/7d1bbe05-289d-414a-8854-db469519e37e-kube-api-access-7fkbq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c665gj4\" (UID: \"7d1bbe05-289d-414a-8854-db469519e37e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c665gj4" Oct 30 00:08:30.608428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.608268 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/712c67e6-6358-4c10-b540-f6f66d889fb0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x4tph\" (UID: \"712c67e6-6358-4c10-b540-f6f66d889fb0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x4tph" Oct 30 00:08:30.608428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.608327 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cmm7\" (UniqueName: \"kubernetes.io/projected/712c67e6-6358-4c10-b540-f6f66d889fb0-kube-api-access-9cmm7\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x4tph\" (UID: \"712c67e6-6358-4c10-b540-f6f66d889fb0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x4tph" Oct 30 00:08:30.608428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.608382 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d1bbe05-289d-414a-8854-db469519e37e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c665gj4\" (UID: \"7d1bbe05-289d-414a-8854-db469519e37e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c665gj4" Oct 30 00:08:30.608677 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.608646 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/712c67e6-6358-4c10-b540-f6f66d889fb0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x4tph\" (UID: \"712c67e6-6358-4c10-b540-f6f66d889fb0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x4tph" Oct 30 00:08:30.608757 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.608709 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d1bbe05-289d-414a-8854-db469519e37e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c665gj4\" (UID: \"7d1bbe05-289d-414a-8854-db469519e37e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c665gj4" Oct 30 00:08:30.608823 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.608774 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d1bbe05-289d-414a-8854-db469519e37e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c665gj4\" (UID: \"7d1bbe05-289d-414a-8854-db469519e37e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c665gj4" Oct 30 00:08:30.609079 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.609056 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d1bbe05-289d-414a-8854-db469519e37e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c665gj4\" (UID: \"7d1bbe05-289d-414a-8854-db469519e37e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c665gj4" Oct 30 00:08:30.643301 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.643266 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fkbq\" (UniqueName: \"kubernetes.io/projected/7d1bbe05-289d-414a-8854-db469519e37e-kube-api-access-7fkbq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c665gj4\" (UID: \"7d1bbe05-289d-414a-8854-db469519e37e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c665gj4" Oct 30 00:08:30.669039 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.669010 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c665gj4" Oct 30 00:08:30.709967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.709766 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/712c67e6-6358-4c10-b540-f6f66d889fb0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x4tph\" (UID: \"712c67e6-6358-4c10-b540-f6f66d889fb0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x4tph" Oct 30 00:08:30.709967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.709843 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cmm7\" (UniqueName: \"kubernetes.io/projected/712c67e6-6358-4c10-b540-f6f66d889fb0-kube-api-access-9cmm7\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x4tph\" (UID: \"712c67e6-6358-4c10-b540-f6f66d889fb0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x4tph" Oct 30 00:08:30.709967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.709882 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08d02afc-16dd-4010-9292-4d394739a984-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5w4t2\" (UID: \"08d02afc-16dd-4010-9292-4d394739a984\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5w4t2" Oct 30 00:08:30.709967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.709940 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbh6p\" (UniqueName: \"kubernetes.io/projected/08d02afc-16dd-4010-9292-4d394739a984-kube-api-access-cbh6p\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5w4t2\" (UID: \"08d02afc-16dd-4010-9292-4d394739a984\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5w4t2" Oct 30 00:08:30.710340 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.710115 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/712c67e6-6358-4c10-b540-f6f66d889fb0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x4tph\" (UID: \"712c67e6-6358-4c10-b540-f6f66d889fb0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x4tph" Oct 30 00:08:30.710340 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.710161 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08d02afc-16dd-4010-9292-4d394739a984-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5w4t2\" (UID: \"08d02afc-16dd-4010-9292-4d394739a984\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5w4t2" Oct 30 00:08:30.710340 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.710194 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/712c67e6-6358-4c10-b540-f6f66d889fb0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x4tph\" (UID: \"712c67e6-6358-4c10-b540-f6f66d889fb0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x4tph" Oct 30 00:08:30.710510 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.710376 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/712c67e6-6358-4c10-b540-f6f66d889fb0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x4tph\" (UID: \"712c67e6-6358-4c10-b540-f6f66d889fb0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x4tph" Oct 30 00:08:30.733979 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.733792 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cmm7\" (UniqueName: \"kubernetes.io/projected/712c67e6-6358-4c10-b540-f6f66d889fb0-kube-api-access-9cmm7\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x4tph\" (UID: \"712c67e6-6358-4c10-b540-f6f66d889fb0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x4tph" Oct 30 00:08:30.767953 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.767924 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x4tph" Oct 30 00:08:30.811068 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.811024 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08d02afc-16dd-4010-9292-4d394739a984-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5w4t2\" (UID: \"08d02afc-16dd-4010-9292-4d394739a984\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5w4t2" Oct 30 00:08:30.811329 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.811092 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbh6p\" (UniqueName: \"kubernetes.io/projected/08d02afc-16dd-4010-9292-4d394739a984-kube-api-access-cbh6p\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5w4t2\" (UID: \"08d02afc-16dd-4010-9292-4d394739a984\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5w4t2" Oct 30 00:08:30.811329 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.811145 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08d02afc-16dd-4010-9292-4d394739a984-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5w4t2\" (UID: \"08d02afc-16dd-4010-9292-4d394739a984\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5w4t2" Oct 30 00:08:30.811614 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.811587 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08d02afc-16dd-4010-9292-4d394739a984-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5w4t2\" (UID: \"08d02afc-16dd-4010-9292-4d394739a984\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5w4t2" Oct 30 00:08:30.811861 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.811836 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08d02afc-16dd-4010-9292-4d394739a984-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5w4t2\" (UID: \"08d02afc-16dd-4010-9292-4d394739a984\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5w4t2" Oct 30 00:08:30.835477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.834690 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbh6p\" (UniqueName: \"kubernetes.io/projected/08d02afc-16dd-4010-9292-4d394739a984-kube-api-access-cbh6p\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5w4t2\" (UID: \"08d02afc-16dd-4010-9292-4d394739a984\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5w4t2" Oct 30 00:08:30.876014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.875983 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c25w57"] Oct 30 00:08:30.877271 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.877245 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5w4t2" Oct 30 00:08:30.880991 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:08:30.880958 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d34d892_f9ab_4bd2_920c_2c7305c7c57b.slice/crio-745af6a21c1e0250ce0199077d629269e76b4d250668ca5aed823e654e57b1ae WatchSource:0}: Error finding container 745af6a21c1e0250ce0199077d629269e76b4d250668ca5aed823e654e57b1ae: Status 404 returned error can't find the container with id 745af6a21c1e0250ce0199077d629269e76b4d250668ca5aed823e654e57b1ae Oct 30 00:08:30.970662 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:30.970627 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c665gj4"] Oct 30 00:08:30.974924 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:08:30.974881 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1bbe05_289d_414a_8854_db469519e37e.slice/crio-9b9b60f52b477fff618468d22a9b5eb3d01c7ebcf9e982e1bef91a39dc022518 WatchSource:0}: Error finding container 9b9b60f52b477fff618468d22a9b5eb3d01c7ebcf9e982e1bef91a39dc022518: Status 404 returned error can't find the container with id 9b9b60f52b477fff618468d22a9b5eb3d01c7ebcf9e982e1bef91a39dc022518 Oct 30 00:08:31.069037 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:31.068947 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x4tph"] Oct 30 00:08:31.073773 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:08:31.073734 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod712c67e6_6358_4c10_b540_f6f66d889fb0.slice/crio-b89c4af907f787da9f4531c4ec5b520b47bc28edf1d871b381425582b8f6f5b4 WatchSource:0}: Error finding container b89c4af907f787da9f4531c4ec5b520b47bc28edf1d871b381425582b8f6f5b4: Status 404 returned error can't find the container with id b89c4af907f787da9f4531c4ec5b520b47bc28edf1d871b381425582b8f6f5b4 Oct 30 00:08:31.179691 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:31.179659 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5w4t2"] Oct 30 00:08:31.184789 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:08:31.184760 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d02afc_16dd_4010_9292_4d394739a984.slice/crio-5361489d895dfcacd14153693a8e556a86734867a5cc42350a298e9c76c8aeb3 WatchSource:0}: Error finding container 5361489d895dfcacd14153693a8e556a86734867a5cc42350a298e9c76c8aeb3: Status 404 returned error can't find the container with id 5361489d895dfcacd14153693a8e556a86734867a5cc42350a298e9c76c8aeb3 Oct 30 00:08:31.692811 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:31.690870 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8c8zc"] Oct 30 00:08:31.695049 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:31.694990 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8c8zc" Oct 30 00:08:31.717508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:31.717479 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-t66mq\"" Oct 30 00:08:31.720795 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:31.720768 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8c8zc"] Oct 30 00:08:31.821699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:31.821658 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c977822-ba67-47f8-9a59-fd265de65217-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8c8zc\" (UID: \"9c977822-ba67-47f8-9a59-fd265de65217\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8c8zc" Oct 30 00:08:31.821893 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:31.821709 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c977822-ba67-47f8-9a59-fd265de65217-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8c8zc\" (UID: \"9c977822-ba67-47f8-9a59-fd265de65217\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8c8zc" Oct 30 00:08:31.821893 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:31.821823 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn4nb\" (UniqueName: \"kubernetes.io/projected/9c977822-ba67-47f8-9a59-fd265de65217-kube-api-access-qn4nb\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8c8zc\" (UID: \"9c977822-ba67-47f8-9a59-fd265de65217\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8c8zc" Oct 30 00:08:31.868421 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:31.868383 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x4tph" event={"ID":"712c67e6-6358-4c10-b540-f6f66d889fb0","Type":"ContainerStarted","Data":"b89c4af907f787da9f4531c4ec5b520b47bc28edf1d871b381425582b8f6f5b4"} Oct 30 00:08:31.870563 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:31.870533 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c25w57" event={"ID":"5d34d892-f9ab-4bd2-920c-2c7305c7c57b","Type":"ContainerStarted","Data":"745af6a21c1e0250ce0199077d629269e76b4d250668ca5aed823e654e57b1ae"} Oct 30 00:08:31.872482 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:31.872426 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5w4t2" event={"ID":"08d02afc-16dd-4010-9292-4d394739a984","Type":"ContainerStarted","Data":"5361489d895dfcacd14153693a8e556a86734867a5cc42350a298e9c76c8aeb3"} Oct 30 00:08:31.875585 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:31.875560 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c665gj4" event={"ID":"7d1bbe05-289d-414a-8854-db469519e37e","Type":"ContainerStarted","Data":"9b9b60f52b477fff618468d22a9b5eb3d01c7ebcf9e982e1bef91a39dc022518"} Oct 30 00:08:31.923287 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:31.923219 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c977822-ba67-47f8-9a59-fd265de65217-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8c8zc\" (UID: \"9c977822-ba67-47f8-9a59-fd265de65217\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8c8zc" Oct 30 00:08:31.923756 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:31.923265 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c977822-ba67-47f8-9a59-fd265de65217-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8c8zc\" (UID: \"9c977822-ba67-47f8-9a59-fd265de65217\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8c8zc" Oct 30 00:08:31.923756 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:31.923353 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn4nb\" (UniqueName: \"kubernetes.io/projected/9c977822-ba67-47f8-9a59-fd265de65217-kube-api-access-qn4nb\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8c8zc\" (UID: \"9c977822-ba67-47f8-9a59-fd265de65217\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8c8zc" Oct 30 00:08:31.924037 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:31.924003 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c977822-ba67-47f8-9a59-fd265de65217-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8c8zc\" (UID: \"9c977822-ba67-47f8-9a59-fd265de65217\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8c8zc" Oct 30 00:08:31.924159 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:31.924032 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c977822-ba67-47f8-9a59-fd265de65217-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8c8zc\" (UID: \"9c977822-ba67-47f8-9a59-fd265de65217\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8c8zc" Oct 30 00:08:31.952280 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:31.952188 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn4nb\" (UniqueName: \"kubernetes.io/projected/9c977822-ba67-47f8-9a59-fd265de65217-kube-api-access-qn4nb\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8c8zc\" (UID: \"9c977822-ba67-47f8-9a59-fd265de65217\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8c8zc" Oct 30 00:08:32.017140 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:32.016891 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8c8zc" Oct 30 00:08:32.353673 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:32.353642 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8c8zc"] Oct 30 00:08:32.359833 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:08:32.359801 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c977822_ba67_47f8_9a59_fd265de65217.slice/crio-ec96b29fcec58f8a9048b190592732c91c8b934d44877d6332319a6e47c47811 WatchSource:0}: Error finding container ec96b29fcec58f8a9048b190592732c91c8b934d44877d6332319a6e47c47811: Status 404 returned error can't find the container with id ec96b29fcec58f8a9048b190592732c91c8b934d44877d6332319a6e47c47811 Oct 30 00:08:32.654605 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:32.654516 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449hnf2"] Oct 30 00:08:32.659878 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:32.659854 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449hnf2" Oct 30 00:08:32.682257 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:32.682228 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-8pbbc\"" Oct 30 00:08:32.684041 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:32.684019 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449hnf2"] Oct 30 00:08:32.832770 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:32.832730 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjwfn\" (UniqueName: \"kubernetes.io/projected/b44f71d9-0521-4d58-a1a4-00aa0b0b11f2-kube-api-access-mjwfn\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449hnf2\" (UID: \"b44f71d9-0521-4d58-a1a4-00aa0b0b11f2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449hnf2" Oct 30 00:08:32.832979 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:32.832840 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b44f71d9-0521-4d58-a1a4-00aa0b0b11f2-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449hnf2\" (UID: \"b44f71d9-0521-4d58-a1a4-00aa0b0b11f2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449hnf2" Oct 30 00:08:32.832979 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:32.832940 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b44f71d9-0521-4d58-a1a4-00aa0b0b11f2-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449hnf2\" (UID: \"b44f71d9-0521-4d58-a1a4-00aa0b0b11f2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449hnf2" Oct 30 00:08:32.887356 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:32.887259 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8c8zc" event={"ID":"9c977822-ba67-47f8-9a59-fd265de65217","Type":"ContainerStarted","Data":"ec96b29fcec58f8a9048b190592732c91c8b934d44877d6332319a6e47c47811"} Oct 30 00:08:32.935028 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:32.934413 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b44f71d9-0521-4d58-a1a4-00aa0b0b11f2-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449hnf2\" (UID: \"b44f71d9-0521-4d58-a1a4-00aa0b0b11f2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449hnf2" Oct 30 00:08:32.935028 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:32.934509 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjwfn\" (UniqueName: \"kubernetes.io/projected/b44f71d9-0521-4d58-a1a4-00aa0b0b11f2-kube-api-access-mjwfn\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449hnf2\" (UID: \"b44f71d9-0521-4d58-a1a4-00aa0b0b11f2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449hnf2" Oct 30 00:08:32.935028 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:32.934544 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b44f71d9-0521-4d58-a1a4-00aa0b0b11f2-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449hnf2\" (UID: \"b44f71d9-0521-4d58-a1a4-00aa0b0b11f2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449hnf2" Oct 30 00:08:32.936413 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:32.936114 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b44f71d9-0521-4d58-a1a4-00aa0b0b11f2-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449hnf2\" (UID: \"b44f71d9-0521-4d58-a1a4-00aa0b0b11f2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449hnf2" Oct 30 00:08:32.936413 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:32.936356 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b44f71d9-0521-4d58-a1a4-00aa0b0b11f2-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449hnf2\" (UID: \"b44f71d9-0521-4d58-a1a4-00aa0b0b11f2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449hnf2" Oct 30 00:08:32.971964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:32.971932 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjwfn\" (UniqueName: \"kubernetes.io/projected/b44f71d9-0521-4d58-a1a4-00aa0b0b11f2-kube-api-access-mjwfn\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449hnf2\" (UID: \"b44f71d9-0521-4d58-a1a4-00aa0b0b11f2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449hnf2" Oct 30 00:08:32.985838 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:32.985808 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449hnf2" Oct 30 00:08:33.343384 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:33.343345 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449hnf2"] Oct 30 00:08:33.896010 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:33.895960 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm"] Oct 30 00:08:33.915597 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:33.915559 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449hnf2" event={"ID":"b44f71d9-0521-4d58-a1a4-00aa0b0b11f2","Type":"ContainerStarted","Data":"c18ee7babcb4149baa27757a1d376939947c022674db5e445d00a4711d155d43"} Oct 30 00:08:33.915770 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:33.915703 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:08:33.928777 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:33.928743 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm"] Oct 30 00:08:33.935891 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:33.935747 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 00:08:33.935891 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:33.935862 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tgqt4\"" Oct 30 00:08:33.945614 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:33.945587 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35bc1599-694f-4f89-9a8a-7667310b9b90-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5sdjm\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:08:33.945764 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:33.945640 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5sdjm\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:08:33.945764 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:33.945668 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5sdjm\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:08:33.945764 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:33.945700 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35bc1599-694f-4f89-9a8a-7667310b9b90-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5sdjm\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:08:33.945764 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:33.945736 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4vwd\" (UniqueName: \"kubernetes.io/projected/35bc1599-694f-4f89-9a8a-7667310b9b90-kube-api-access-v4vwd\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5sdjm\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:08:33.946119 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:33.945782 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5sdjm\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:08:33.946119 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:33.945845 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5sdjm\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:08:33.946119 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:33.945884 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5sdjm\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:08:33.946119 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:33.945937 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35bc1599-694f-4f89-9a8a-7667310b9b90-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5sdjm\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:08:34.025427 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.023840 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh"] Oct 30 00:08:34.028259 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.028239 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" Oct 30 00:08:34.046313 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.046277 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5sdjm\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:08:34.046447 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.046325 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5sdjm\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:08:34.046447 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.046356 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35bc1599-694f-4f89-9a8a-7667310b9b90-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5sdjm\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:08:34.046447 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.046392 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hccqm\" (UniqueName: \"kubernetes.io/projected/f9b032e2-dad9-4256-a988-8e6020b4da43-kube-api-access-hccqm\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh\" (UID: \"f9b032e2-dad9-4256-a988-8e6020b4da43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" Oct 30 00:08:34.046447 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.046424 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4vwd\" (UniqueName: \"kubernetes.io/projected/35bc1599-694f-4f89-9a8a-7667310b9b90-kube-api-access-v4vwd\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5sdjm\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:08:34.046780 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.046477 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5sdjm\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:08:34.046780 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.046534 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b032e2-dad9-4256-a988-8e6020b4da43-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh\" (UID: \"f9b032e2-dad9-4256-a988-8e6020b4da43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" Oct 30 00:08:34.046780 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.046560 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5sdjm\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:08:34.046780 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.046581 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b032e2-dad9-4256-a988-8e6020b4da43-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh\" (UID: \"f9b032e2-dad9-4256-a988-8e6020b4da43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" Oct 30 00:08:34.046780 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.046596 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b032e2-dad9-4256-a988-8e6020b4da43-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh\" (UID: \"f9b032e2-dad9-4256-a988-8e6020b4da43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" Oct 30 00:08:34.046780 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.046621 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5sdjm\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:08:34.046780 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.046643 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b032e2-dad9-4256-a988-8e6020b4da43-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh\" (UID: \"f9b032e2-dad9-4256-a988-8e6020b4da43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" Oct 30 00:08:34.046780 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.046659 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b032e2-dad9-4256-a988-8e6020b4da43-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh\" (UID: \"f9b032e2-dad9-4256-a988-8e6020b4da43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" Oct 30 00:08:34.046780 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.046676 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35bc1599-694f-4f89-9a8a-7667310b9b90-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5sdjm\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:08:34.046780 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.046709 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b032e2-dad9-4256-a988-8e6020b4da43-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh\" (UID: \"f9b032e2-dad9-4256-a988-8e6020b4da43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" Oct 30 00:08:34.046780 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.046724 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b032e2-dad9-4256-a988-8e6020b4da43-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh\" (UID: \"f9b032e2-dad9-4256-a988-8e6020b4da43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" Oct 30 00:08:34.046780 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.046741 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b032e2-dad9-4256-a988-8e6020b4da43-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh\" (UID: \"f9b032e2-dad9-4256-a988-8e6020b4da43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" Oct 30 00:08:34.046780 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.046764 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35bc1599-694f-4f89-9a8a-7667310b9b90-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5sdjm\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:08:34.049025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.048610 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35bc1599-694f-4f89-9a8a-7667310b9b90-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5sdjm\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:08:34.049025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.048697 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5sdjm\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:08:34.049025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.048986 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5sdjm\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:08:34.051055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.050231 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5sdjm\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:08:34.051055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.050853 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5sdjm\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:08:34.056463 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.056350 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh"] Oct 30 00:08:34.078570 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.078228 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5sdjm\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:08:34.079378 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.079324 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35bc1599-694f-4f89-9a8a-7667310b9b90-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5sdjm\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:08:34.087631 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.087585 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35bc1599-694f-4f89-9a8a-7667310b9b90-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5sdjm\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:08:34.088940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.088905 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4vwd\" (UniqueName: \"kubernetes.io/projected/35bc1599-694f-4f89-9a8a-7667310b9b90-kube-api-access-v4vwd\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5sdjm\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:08:34.130807 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.130747 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-vq7p9\"" Oct 30 00:08:34.147801 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.147721 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b032e2-dad9-4256-a988-8e6020b4da43-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh\" (UID: \"f9b032e2-dad9-4256-a988-8e6020b4da43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" Oct 30 00:08:34.147801 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.147771 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b032e2-dad9-4256-a988-8e6020b4da43-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh\" (UID: \"f9b032e2-dad9-4256-a988-8e6020b4da43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" Oct 30 00:08:34.148116 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.147802 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b032e2-dad9-4256-a988-8e6020b4da43-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh\" (UID: \"f9b032e2-dad9-4256-a988-8e6020b4da43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" Oct 30 00:08:34.148116 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.147857 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hccqm\" (UniqueName: \"kubernetes.io/projected/f9b032e2-dad9-4256-a988-8e6020b4da43-kube-api-access-hccqm\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh\" (UID: \"f9b032e2-dad9-4256-a988-8e6020b4da43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" Oct 30 00:08:34.148116 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.147963 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b032e2-dad9-4256-a988-8e6020b4da43-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh\" (UID: \"f9b032e2-dad9-4256-a988-8e6020b4da43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" Oct 30 00:08:34.148116 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.147995 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b032e2-dad9-4256-a988-8e6020b4da43-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh\" (UID: \"f9b032e2-dad9-4256-a988-8e6020b4da43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" Oct 30 00:08:34.148116 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.148018 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b032e2-dad9-4256-a988-8e6020b4da43-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh\" (UID: \"f9b032e2-dad9-4256-a988-8e6020b4da43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" Oct 30 00:08:34.148116 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.148056 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b032e2-dad9-4256-a988-8e6020b4da43-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh\" (UID: \"f9b032e2-dad9-4256-a988-8e6020b4da43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" Oct 30 00:08:34.148116 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.148080 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b032e2-dad9-4256-a988-8e6020b4da43-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh\" (UID: \"f9b032e2-dad9-4256-a988-8e6020b4da43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" Oct 30 00:08:34.148646 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.148619 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b032e2-dad9-4256-a988-8e6020b4da43-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh\" (UID: \"f9b032e2-dad9-4256-a988-8e6020b4da43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" Oct 30 00:08:34.149485 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.149310 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b032e2-dad9-4256-a988-8e6020b4da43-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh\" (UID: \"f9b032e2-dad9-4256-a988-8e6020b4da43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" Oct 30 00:08:34.149485 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.149448 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b032e2-dad9-4256-a988-8e6020b4da43-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh\" (UID: \"f9b032e2-dad9-4256-a988-8e6020b4da43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" Oct 30 00:08:34.149651 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.149500 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b032e2-dad9-4256-a988-8e6020b4da43-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh\" (UID: \"f9b032e2-dad9-4256-a988-8e6020b4da43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" Oct 30 00:08:34.149965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.149939 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b032e2-dad9-4256-a988-8e6020b4da43-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh\" (UID: \"f9b032e2-dad9-4256-a988-8e6020b4da43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" Oct 30 00:08:34.157245 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.157195 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b032e2-dad9-4256-a988-8e6020b4da43-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh\" (UID: \"f9b032e2-dad9-4256-a988-8e6020b4da43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" Oct 30 00:08:34.166523 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.166498 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b032e2-dad9-4256-a988-8e6020b4da43-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh\" (UID: \"f9b032e2-dad9-4256-a988-8e6020b4da43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" Oct 30 00:08:34.178453 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.177187 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b032e2-dad9-4256-a988-8e6020b4da43-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh\" (UID: \"f9b032e2-dad9-4256-a988-8e6020b4da43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" Oct 30 00:08:34.184996 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.184965 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hccqm\" (UniqueName: \"kubernetes.io/projected/f9b032e2-dad9-4256-a988-8e6020b4da43-kube-api-access-hccqm\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh\" (UID: \"f9b032e2-dad9-4256-a988-8e6020b4da43\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" Oct 30 00:08:34.236581 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.236545 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:08:34.354037 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.353673 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" Oct 30 00:08:34.611081 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.611049 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm"] Oct 30 00:08:34.618540 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:08:34.618495 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35bc1599_694f_4f89_9a8a_7667310b9b90.slice/crio-0c6d45180e45a143c757b0815ba0da9a85a78cbb8ce6a40c5961e3ef06480df5 WatchSource:0}: Error finding container 0c6d45180e45a143c757b0815ba0da9a85a78cbb8ce6a40c5961e3ef06480df5: Status 404 returned error can't find the container with id 0c6d45180e45a143c757b0815ba0da9a85a78cbb8ce6a40c5961e3ef06480df5 Oct 30 00:08:34.776855 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.776819 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh"] Oct 30 00:08:34.789835 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:08:34.789798 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b032e2_dad9_4256_a988_8e6020b4da43.slice/crio-1a7cd7a00718e1d347f584f48a32771c19dbabbb0afb502b3a4c26fb6a68bfd9 WatchSource:0}: Error finding container 1a7cd7a00718e1d347f584f48a32771c19dbabbb0afb502b3a4c26fb6a68bfd9: Status 404 returned error can't find the container with id 1a7cd7a00718e1d347f584f48a32771c19dbabbb0afb502b3a4c26fb6a68bfd9 Oct 30 00:08:34.928322 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.928284 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" event={"ID":"35bc1599-694f-4f89-9a8a-7667310b9b90","Type":"ContainerStarted","Data":"0c6d45180e45a143c757b0815ba0da9a85a78cbb8ce6a40c5961e3ef06480df5"} Oct 30 00:08:34.928504 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:34.928342 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" event={"ID":"f9b032e2-dad9-4256-a988-8e6020b4da43","Type":"ContainerStarted","Data":"1a7cd7a00718e1d347f584f48a32771c19dbabbb0afb502b3a4c26fb6a68bfd9"} Oct 30 00:08:38.016835 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.016799 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxvr2f"] Oct 30 00:08:38.021606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.021583 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxvr2f" Oct 30 00:08:38.046311 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.046172 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxvr2f"] Oct 30 00:08:38.056778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.054853 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dlg25"] Oct 30 00:08:38.059675 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.059142 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dlg25" Oct 30 00:08:38.082595 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.082525 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dlg25"] Oct 30 00:08:38.084827 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.084805 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atzn5p"] Oct 30 00:08:38.089134 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.089111 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atzn5p" Oct 30 00:08:38.091041 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.091014 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/95276631-3e7c-4d4b-a182-8b5d0fa319b8-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxvr2f\" (UID: \"95276631-3e7c-4d4b-a182-8b5d0fa319b8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxvr2f" Oct 30 00:08:38.091144 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.091066 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ljtz\" (UniqueName: \"kubernetes.io/projected/95276631-3e7c-4d4b-a182-8b5d0fa319b8-kube-api-access-7ljtz\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxvr2f\" (UID: \"95276631-3e7c-4d4b-a182-8b5d0fa319b8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxvr2f" Oct 30 00:08:38.091208 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.091192 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a78c8440-3039-46ab-b3c6-d0276d984f56-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dlg25\" (UID: \"a78c8440-3039-46ab-b3c6-d0276d984f56\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dlg25" Oct 30 00:08:38.091267 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.091234 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a78c8440-3039-46ab-b3c6-d0276d984f56-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dlg25\" (UID: \"a78c8440-3039-46ab-b3c6-d0276d984f56\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dlg25" Oct 30 00:08:38.091321 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.091264 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5fgl\" (UniqueName: \"kubernetes.io/projected/a78c8440-3039-46ab-b3c6-d0276d984f56-kube-api-access-b5fgl\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dlg25\" (UID: \"a78c8440-3039-46ab-b3c6-d0276d984f56\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dlg25" Oct 30 00:08:38.091321 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.091294 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/95276631-3e7c-4d4b-a182-8b5d0fa319b8-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxvr2f\" (UID: \"95276631-3e7c-4d4b-a182-8b5d0fa319b8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxvr2f" Oct 30 00:08:38.114863 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.113970 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atzn5p"] Oct 30 00:08:38.114863 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.114696 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pvbrf"] Oct 30 00:08:38.119655 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.119637 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pvbrf" Oct 30 00:08:38.138767 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.138710 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pvbrf"] Oct 30 00:08:38.194634 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.192743 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a78c8440-3039-46ab-b3c6-d0276d984f56-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dlg25\" (UID: \"a78c8440-3039-46ab-b3c6-d0276d984f56\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dlg25" Oct 30 00:08:38.194634 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.192796 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbffa049-445d-47f2-bcf6-6edd14cf25c1-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atzn5p\" (UID: \"bbffa049-445d-47f2-bcf6-6edd14cf25c1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atzn5p" Oct 30 00:08:38.194634 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.192830 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xsjj\" (UniqueName: \"kubernetes.io/projected/556143f7-0f4b-456c-98c1-da699bc3c9c3-kube-api-access-2xsjj\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pvbrf\" (UID: \"556143f7-0f4b-456c-98c1-da699bc3c9c3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pvbrf" Oct 30 00:08:38.194634 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.192859 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a78c8440-3039-46ab-b3c6-d0276d984f56-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dlg25\" (UID: \"a78c8440-3039-46ab-b3c6-d0276d984f56\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dlg25" Oct 30 00:08:38.194634 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.192890 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5fgl\" (UniqueName: \"kubernetes.io/projected/a78c8440-3039-46ab-b3c6-d0276d984f56-kube-api-access-b5fgl\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dlg25\" (UID: \"a78c8440-3039-46ab-b3c6-d0276d984f56\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dlg25" Oct 30 00:08:38.194634 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.192930 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbffa049-445d-47f2-bcf6-6edd14cf25c1-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atzn5p\" (UID: \"bbffa049-445d-47f2-bcf6-6edd14cf25c1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atzn5p" Oct 30 00:08:38.194634 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.192961 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/95276631-3e7c-4d4b-a182-8b5d0fa319b8-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxvr2f\" (UID: \"95276631-3e7c-4d4b-a182-8b5d0fa319b8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxvr2f" Oct 30 00:08:38.194634 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.192999 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/556143f7-0f4b-456c-98c1-da699bc3c9c3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pvbrf\" (UID: \"556143f7-0f4b-456c-98c1-da699bc3c9c3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pvbrf" Oct 30 00:08:38.194634 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.193031 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/556143f7-0f4b-456c-98c1-da699bc3c9c3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pvbrf\" (UID: \"556143f7-0f4b-456c-98c1-da699bc3c9c3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pvbrf" Oct 30 00:08:38.194634 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.193050 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h7ct\" (UniqueName: \"kubernetes.io/projected/bbffa049-445d-47f2-bcf6-6edd14cf25c1-kube-api-access-7h7ct\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atzn5p\" (UID: \"bbffa049-445d-47f2-bcf6-6edd14cf25c1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atzn5p" Oct 30 00:08:38.194634 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.193083 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/95276631-3e7c-4d4b-a182-8b5d0fa319b8-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxvr2f\" (UID: \"95276631-3e7c-4d4b-a182-8b5d0fa319b8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxvr2f" Oct 30 00:08:38.194634 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.193097 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ljtz\" (UniqueName: \"kubernetes.io/projected/95276631-3e7c-4d4b-a182-8b5d0fa319b8-kube-api-access-7ljtz\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxvr2f\" (UID: \"95276631-3e7c-4d4b-a182-8b5d0fa319b8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxvr2f" Oct 30 00:08:38.194634 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.193615 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a78c8440-3039-46ab-b3c6-d0276d984f56-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dlg25\" (UID: \"a78c8440-3039-46ab-b3c6-d0276d984f56\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dlg25" Oct 30 00:08:38.194634 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.193802 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a78c8440-3039-46ab-b3c6-d0276d984f56-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dlg25\" (UID: \"a78c8440-3039-46ab-b3c6-d0276d984f56\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dlg25" Oct 30 00:08:38.194634 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.194323 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/95276631-3e7c-4d4b-a182-8b5d0fa319b8-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxvr2f\" (UID: \"95276631-3e7c-4d4b-a182-8b5d0fa319b8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxvr2f" Oct 30 00:08:38.194634 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.194589 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/95276631-3e7c-4d4b-a182-8b5d0fa319b8-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxvr2f\" (UID: \"95276631-3e7c-4d4b-a182-8b5d0fa319b8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxvr2f" Oct 30 00:08:38.233215 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.233153 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5fgl\" (UniqueName: \"kubernetes.io/projected/a78c8440-3039-46ab-b3c6-d0276d984f56-kube-api-access-b5fgl\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dlg25\" (UID: \"a78c8440-3039-46ab-b3c6-d0276d984f56\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dlg25" Oct 30 00:08:38.233851 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.233804 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ljtz\" (UniqueName: \"kubernetes.io/projected/95276631-3e7c-4d4b-a182-8b5d0fa319b8-kube-api-access-7ljtz\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxvr2f\" (UID: \"95276631-3e7c-4d4b-a182-8b5d0fa319b8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxvr2f" Oct 30 00:08:38.294630 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.294521 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbffa049-445d-47f2-bcf6-6edd14cf25c1-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atzn5p\" (UID: \"bbffa049-445d-47f2-bcf6-6edd14cf25c1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atzn5p" Oct 30 00:08:38.294630 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.294604 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xsjj\" (UniqueName: \"kubernetes.io/projected/556143f7-0f4b-456c-98c1-da699bc3c9c3-kube-api-access-2xsjj\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pvbrf\" (UID: \"556143f7-0f4b-456c-98c1-da699bc3c9c3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pvbrf" Oct 30 00:08:38.294901 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.294649 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbffa049-445d-47f2-bcf6-6edd14cf25c1-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atzn5p\" (UID: \"bbffa049-445d-47f2-bcf6-6edd14cf25c1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atzn5p" Oct 30 00:08:38.294901 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.294693 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/556143f7-0f4b-456c-98c1-da699bc3c9c3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pvbrf\" (UID: \"556143f7-0f4b-456c-98c1-da699bc3c9c3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pvbrf" Oct 30 00:08:38.294901 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.294738 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/556143f7-0f4b-456c-98c1-da699bc3c9c3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pvbrf\" (UID: \"556143f7-0f4b-456c-98c1-da699bc3c9c3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pvbrf" Oct 30 00:08:38.294901 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.294771 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h7ct\" (UniqueName: \"kubernetes.io/projected/bbffa049-445d-47f2-bcf6-6edd14cf25c1-kube-api-access-7h7ct\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atzn5p\" (UID: \"bbffa049-445d-47f2-bcf6-6edd14cf25c1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atzn5p" Oct 30 00:08:38.295548 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.295522 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbffa049-445d-47f2-bcf6-6edd14cf25c1-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atzn5p\" (UID: \"bbffa049-445d-47f2-bcf6-6edd14cf25c1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atzn5p" Oct 30 00:08:38.296065 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.296005 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbffa049-445d-47f2-bcf6-6edd14cf25c1-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atzn5p\" (UID: \"bbffa049-445d-47f2-bcf6-6edd14cf25c1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atzn5p" Oct 30 00:08:38.296322 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.296301 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/556143f7-0f4b-456c-98c1-da699bc3c9c3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pvbrf\" (UID: \"556143f7-0f4b-456c-98c1-da699bc3c9c3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pvbrf" Oct 30 00:08:38.296550 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.296530 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/556143f7-0f4b-456c-98c1-da699bc3c9c3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pvbrf\" (UID: \"556143f7-0f4b-456c-98c1-da699bc3c9c3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pvbrf" Oct 30 00:08:38.338605 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.338573 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h7ct\" (UniqueName: \"kubernetes.io/projected/bbffa049-445d-47f2-bcf6-6edd14cf25c1-kube-api-access-7h7ct\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atzn5p\" (UID: \"bbffa049-445d-47f2-bcf6-6edd14cf25c1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atzn5p" Oct 30 00:08:38.347121 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.347097 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxvr2f" Oct 30 00:08:38.368991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.368884 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xsjj\" (UniqueName: \"kubernetes.io/projected/556143f7-0f4b-456c-98c1-da699bc3c9c3-kube-api-access-2xsjj\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pvbrf\" (UID: \"556143f7-0f4b-456c-98c1-da699bc3c9c3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pvbrf" Oct 30 00:08:38.388539 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.388512 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dlg25" Oct 30 00:08:38.426999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.426730 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atzn5p" Oct 30 00:08:38.445461 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.445231 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pvbrf" Oct 30 00:08:38.769828 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.769700 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dlg25"] Oct 30 00:08:38.780960 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.780862 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxvr2f"] Oct 30 00:08:38.783538 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.783481 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atzn5p"] Oct 30 00:08:38.797032 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:08:38.796995 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbffa049_445d_47f2_bcf6_6edd14cf25c1.slice/crio-a71f5451df2846f1032b846f80285c231c907657fda1f7f6e236a2977ebac98f WatchSource:0}: Error finding container a71f5451df2846f1032b846f80285c231c907657fda1f7f6e236a2977ebac98f: Status 404 returned error can't find the container with id a71f5451df2846f1032b846f80285c231c907657fda1f7f6e236a2977ebac98f Oct 30 00:08:38.827214 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.827141 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pvbrf"] Oct 30 00:08:38.836134 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:08:38.836099 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod556143f7_0f4b_456c_98c1_da699bc3c9c3.slice/crio-46235b152c7ac4079d10ab1eb19218af2501735956c32de5da08553ce2e82f23 WatchSource:0}: Error finding container 46235b152c7ac4079d10ab1eb19218af2501735956c32de5da08553ce2e82f23: Status 404 returned error can't find the container with id 46235b152c7ac4079d10ab1eb19218af2501735956c32de5da08553ce2e82f23 Oct 30 00:08:38.987630 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.987586 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dlg25" event={"ID":"a78c8440-3039-46ab-b3c6-d0276d984f56","Type":"ContainerStarted","Data":"711e6c41c60977f6617ff1940ac6901763e10355280f2bee496d02091051a664"} Oct 30 00:08:38.992697 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:38.992588 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pvbrf" event={"ID":"556143f7-0f4b-456c-98c1-da699bc3c9c3","Type":"ContainerStarted","Data":"46235b152c7ac4079d10ab1eb19218af2501735956c32de5da08553ce2e82f23"} Oct 30 00:08:39.005188 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:39.005093 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atzn5p" event={"ID":"bbffa049-445d-47f2-bcf6-6edd14cf25c1","Type":"ContainerStarted","Data":"a71f5451df2846f1032b846f80285c231c907657fda1f7f6e236a2977ebac98f"} Oct 30 00:08:39.010350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:39.010272 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxvr2f" event={"ID":"95276631-3e7c-4d4b-a182-8b5d0fa319b8","Type":"ContainerStarted","Data":"6771d7d501bf494e8c6693753964b21f519cf0f1598b0d10524905b7b57e0a52"} Oct 30 00:08:45.111117 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:45.110261 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:08:45.111117 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:45.110350 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:08:45.111117 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:45.110393 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:08:45.145271 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:45.145133 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:08:45.145271 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:45.145221 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:08:45.145437 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:45.145272 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:08:46.109984 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:46.109022 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" event={"ID":"f9b032e2-dad9-4256-a988-8e6020b4da43","Type":"ContainerStarted","Data":"7c76716cbe5ba72fa3144099025b555bca48c587d16fde5a4becf302189fcd13"} Oct 30 00:08:46.121086 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:46.121050 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" event={"ID":"35bc1599-694f-4f89-9a8a-7667310b9b90","Type":"ContainerStarted","Data":"f729a1e075e62ccca8b57280d27c2a18975fbeaf902c4c76fae53e6e51fd0d2b"} Oct 30 00:08:46.209503 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:46.209439 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" podStartSLOduration=2.898249424 podStartE2EDuration="13.209395438s" totalImagesPullingTime="10.311146014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:08:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:08:46.165995143 +0000 UTC m=+241.371496349" watchObservedRunningTime="2026-10-30 00:08:46.209395438 +0000 UTC m=+241.414896621" Oct 30 00:08:46.238903 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:46.238871 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:08:46.243319 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:46.243291 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:08:46.283897 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:46.283841 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" podStartSLOduration=2.761919915 podStartE2EDuration="13.283823219s" totalImagesPullingTime="10.521903304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:08:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:08:46.219117199 +0000 UTC m=+241.424618417" watchObservedRunningTime="2026-10-30 00:08:46.283823219 +0000 UTC m=+241.489324403" Oct 30 00:08:46.354782 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:46.354746 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" Oct 30 00:08:46.361390 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:46.361295 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" Oct 30 00:08:47.146337 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:47.143970 5001 generic.go:356] "Generic (PLEG): container finished" podID="4484ecc5-88d4-4f8e-8244-a7b35f95bb3a" containerID="a6cf980e3f21df437ca981367d86d3688689470df1302cd539130d14a512aa9a" exitCode=0 Oct 30 00:08:47.146337 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:47.144161 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716pmws" event={"ID":"4484ecc5-88d4-4f8e-8244-a7b35f95bb3a","Type":"ContainerDied","Data":"a6cf980e3f21df437ca981367d86d3688689470df1302cd539130d14a512aa9a"} Oct 30 00:08:47.146337 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:47.146170 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:08:47.146337 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:47.146226 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" Oct 30 00:08:47.147506 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:47.147415 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh" Oct 30 00:08:47.148220 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:47.148196 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:08:47.365373 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:08:47.365307 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:08:49.161106 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:49.161059 5001 generic.go:356] "Generic (PLEG): container finished" podID="4484ecc5-88d4-4f8e-8244-a7b35f95bb3a" containerID="dae2704237ac33f9132de6e41e0e677a7a7f9af095122ca1a7d9119c9fca9c0b" exitCode=0 Oct 30 00:08:49.162363 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:49.162336 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716pmws" event={"ID":"4484ecc5-88d4-4f8e-8244-a7b35f95bb3a","Type":"ContainerDied","Data":"dae2704237ac33f9132de6e41e0e677a7a7f9af095122ca1a7d9119c9fca9c0b"} Oct 30 00:08:51.187973 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:51.187172 5001 generic.go:356] "Generic (PLEG): container finished" podID="5d34d892-f9ab-4bd2-920c-2c7305c7c57b" containerID="93e78b9a1b204c1fecdfc87e207c1e953dd043b13ee923808528626cb2406304" exitCode=0 Oct 30 00:08:51.187973 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:51.187259 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c25w57" event={"ID":"5d34d892-f9ab-4bd2-920c-2c7305c7c57b","Type":"ContainerDied","Data":"93e78b9a1b204c1fecdfc87e207c1e953dd043b13ee923808528626cb2406304"} Oct 30 00:08:52.196058 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:52.196013 5001 generic.go:356] "Generic (PLEG): container finished" podID="08d02afc-16dd-4010-9292-4d394739a984" containerID="c8ba8cf312cacb28a09452f5745d05ba5c42763e2d78a5620f78dcc34abeb6fc" exitCode=0 Oct 30 00:08:52.196554 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:52.196096 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5w4t2" event={"ID":"08d02afc-16dd-4010-9292-4d394739a984","Type":"ContainerDied","Data":"c8ba8cf312cacb28a09452f5745d05ba5c42763e2d78a5620f78dcc34abeb6fc"} Oct 30 00:08:52.200069 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:52.200000 5001 generic.go:356] "Generic (PLEG): container finished" podID="7d1bbe05-289d-414a-8854-db469519e37e" containerID="796ecaccc3c387245b6fc0babf22a953688c239b76e29f9d65193a3bfadab923" exitCode=0 Oct 30 00:08:52.200126 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:52.200079 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c665gj4" event={"ID":"7d1bbe05-289d-414a-8854-db469519e37e","Type":"ContainerDied","Data":"796ecaccc3c387245b6fc0babf22a953688c239b76e29f9d65193a3bfadab923"} Oct 30 00:08:52.202959 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:52.202935 5001 generic.go:356] "Generic (PLEG): container finished" podID="a78c8440-3039-46ab-b3c6-d0276d984f56" containerID="344ec0a7b8da0cb7f052e53318d044ee1508317b803dc2658697824841cf65b2" exitCode=0 Oct 30 00:08:52.203081 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:52.203067 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dlg25" event={"ID":"a78c8440-3039-46ab-b3c6-d0276d984f56","Type":"ContainerDied","Data":"344ec0a7b8da0cb7f052e53318d044ee1508317b803dc2658697824841cf65b2"} Oct 30 00:08:53.209375 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:53.209337 5001 generic.go:356] "Generic (PLEG): container finished" podID="08d02afc-16dd-4010-9292-4d394739a984" containerID="046fa39295e784b27acb8c5637fb5ce5542a4a5545aaa5858e0d12168ff3167b" exitCode=0 Oct 30 00:08:53.209805 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:53.209474 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5w4t2" event={"ID":"08d02afc-16dd-4010-9292-4d394739a984","Type":"ContainerDied","Data":"046fa39295e784b27acb8c5637fb5ce5542a4a5545aaa5858e0d12168ff3167b"} Oct 30 00:08:53.211787 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:53.211249 5001 generic.go:356] "Generic (PLEG): container finished" podID="556143f7-0f4b-456c-98c1-da699bc3c9c3" containerID="26aed424eaefaed7f91608818e973c6653295ecc979f9c619ffffb7e996d5f25" exitCode=0 Oct 30 00:08:53.211787 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:53.211363 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pvbrf" event={"ID":"556143f7-0f4b-456c-98c1-da699bc3c9c3","Type":"ContainerDied","Data":"26aed424eaefaed7f91608818e973c6653295ecc979f9c619ffffb7e996d5f25"} Oct 30 00:08:53.213410 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:53.213386 5001 generic.go:356] "Generic (PLEG): container finished" podID="9c977822-ba67-47f8-9a59-fd265de65217" containerID="eba309c4f6449701ae4f6f323820fb00066a52c88e24c5cc0d32dfee91811911" exitCode=0 Oct 30 00:08:53.213514 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:53.213465 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8c8zc" event={"ID":"9c977822-ba67-47f8-9a59-fd265de65217","Type":"ContainerDied","Data":"eba309c4f6449701ae4f6f323820fb00066a52c88e24c5cc0d32dfee91811911"} Oct 30 00:08:53.216495 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:53.216474 5001 generic.go:356] "Generic (PLEG): container finished" podID="bbffa049-445d-47f2-bcf6-6edd14cf25c1" containerID="33394b2b4505ebf97bbd216b5ef7e80308edb74528c69c8db43a2f09ed43a4e2" exitCode=0 Oct 30 00:08:53.217000 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:53.216976 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atzn5p" event={"ID":"bbffa049-445d-47f2-bcf6-6edd14cf25c1","Type":"ContainerDied","Data":"33394b2b4505ebf97bbd216b5ef7e80308edb74528c69c8db43a2f09ed43a4e2"} Oct 30 00:08:53.219476 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:53.219457 5001 generic.go:356] "Generic (PLEG): container finished" podID="95276631-3e7c-4d4b-a182-8b5d0fa319b8" containerID="dbfed336053419fd026abd6a2e7361dbb472b6c37e8299aa4c248ea5364f790f" exitCode=0 Oct 30 00:08:53.219565 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:53.219486 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxvr2f" event={"ID":"95276631-3e7c-4d4b-a182-8b5d0fa319b8","Type":"ContainerDied","Data":"dbfed336053419fd026abd6a2e7361dbb472b6c37e8299aa4c248ea5364f790f"} Oct 30 00:08:53.221382 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:53.221304 5001 generic.go:356] "Generic (PLEG): container finished" podID="712c67e6-6358-4c10-b540-f6f66d889fb0" containerID="050164c38cdd42f68ceb3250c3608c7bd77d738b1c9d47ac4189fbaf697c131e" exitCode=0 Oct 30 00:08:53.221853 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:53.221833 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x4tph" event={"ID":"712c67e6-6358-4c10-b540-f6f66d889fb0","Type":"ContainerDied","Data":"050164c38cdd42f68ceb3250c3608c7bd77d738b1c9d47ac4189fbaf697c131e"} Oct 30 00:08:53.225167 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:53.225117 5001 generic.go:356] "Generic (PLEG): container finished" podID="5d34d892-f9ab-4bd2-920c-2c7305c7c57b" containerID="d80a760f761eef6a663d5c19d44490c1e358cf17d8bd0aba4de6e1d4d1a8eddc" exitCode=0 Oct 30 00:08:53.225279 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:53.225222 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c25w57" event={"ID":"5d34d892-f9ab-4bd2-920c-2c7305c7c57b","Type":"ContainerDied","Data":"d80a760f761eef6a663d5c19d44490c1e358cf17d8bd0aba4de6e1d4d1a8eddc"} Oct 30 00:08:53.228204 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:53.228181 5001 generic.go:356] "Generic (PLEG): container finished" podID="b44f71d9-0521-4d58-a1a4-00aa0b0b11f2" containerID="dd0cbc2b30dbb387411da89555127f53baca4656e8c70faf6e973ef7ba053c2d" exitCode=0 Oct 30 00:08:53.228461 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:53.228295 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449hnf2" event={"ID":"b44f71d9-0521-4d58-a1a4-00aa0b0b11f2","Type":"ContainerDied","Data":"dd0cbc2b30dbb387411da89555127f53baca4656e8c70faf6e973ef7ba053c2d"} Oct 30 00:08:54.248107 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:54.248035 5001 generic.go:356] "Generic (PLEG): container finished" podID="bbffa049-445d-47f2-bcf6-6edd14cf25c1" containerID="16949760cf15260a2033137914620f8ef20765cdb643fd06fcafcc32d12b405f" exitCode=0 Oct 30 00:08:54.248510 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:54.248142 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atzn5p" event={"ID":"bbffa049-445d-47f2-bcf6-6edd14cf25c1","Type":"ContainerDied","Data":"16949760cf15260a2033137914620f8ef20765cdb643fd06fcafcc32d12b405f"} Oct 30 00:08:54.253897 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:54.253847 5001 generic.go:356] "Generic (PLEG): container finished" podID="95276631-3e7c-4d4b-a182-8b5d0fa319b8" containerID="fd8564fa86c130e05b52ee7fc2bbfc999a906e69d9e0c98294eafab5298da183" exitCode=0 Oct 30 00:08:54.254029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:54.254009 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxvr2f" event={"ID":"95276631-3e7c-4d4b-a182-8b5d0fa319b8","Type":"ContainerDied","Data":"fd8564fa86c130e05b52ee7fc2bbfc999a906e69d9e0c98294eafab5298da183"} Oct 30 00:08:54.261494 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:54.260249 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x4tph" event={"ID":"712c67e6-6358-4c10-b540-f6f66d889fb0","Type":"ContainerStarted","Data":"39632d04299a346a63414f5a6db3831a6410983c766d382e4112ff23ec0e64ea"} Oct 30 00:08:54.263813 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:54.263790 5001 generic.go:356] "Generic (PLEG): container finished" podID="7d1bbe05-289d-414a-8854-db469519e37e" containerID="278435d974cd3093d388593bfec1e14a383b4fa63ee24a0a87f2b1045e6e8f51" exitCode=0 Oct 30 00:08:54.263940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:54.263868 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c665gj4" event={"ID":"7d1bbe05-289d-414a-8854-db469519e37e","Type":"ContainerDied","Data":"278435d974cd3093d388593bfec1e14a383b4fa63ee24a0a87f2b1045e6e8f51"} Oct 30 00:08:54.267443 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:54.267424 5001 generic.go:356] "Generic (PLEG): container finished" podID="a78c8440-3039-46ab-b3c6-d0276d984f56" containerID="eeda7547b83469f0d4bdda64bae85e9a43e9529f637bb174adf1ffc2e1b9dd48" exitCode=0 Oct 30 00:08:54.267652 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:54.267602 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dlg25" event={"ID":"a78c8440-3039-46ab-b3c6-d0276d984f56","Type":"ContainerDied","Data":"eeda7547b83469f0d4bdda64bae85e9a43e9529f637bb174adf1ffc2e1b9dd48"} Oct 30 00:08:54.277583 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:54.277562 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pvbrf" event={"ID":"556143f7-0f4b-456c-98c1-da699bc3c9c3","Type":"ContainerStarted","Data":"8dbf163803bbc5130c03990d3d479fb2f848183e04a34b8e0d8d6c4dd1756b54"} Oct 30 00:08:55.289844 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:55.289634 5001 generic.go:356] "Generic (PLEG): container finished" podID="556143f7-0f4b-456c-98c1-da699bc3c9c3" containerID="8dbf163803bbc5130c03990d3d479fb2f848183e04a34b8e0d8d6c4dd1756b54" exitCode=0 Oct 30 00:08:55.289844 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:55.289776 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pvbrf" event={"ID":"556143f7-0f4b-456c-98c1-da699bc3c9c3","Type":"ContainerDied","Data":"8dbf163803bbc5130c03990d3d479fb2f848183e04a34b8e0d8d6c4dd1756b54"} Oct 30 00:08:55.293266 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:55.293230 5001 generic.go:356] "Generic (PLEG): container finished" podID="712c67e6-6358-4c10-b540-f6f66d889fb0" containerID="39632d04299a346a63414f5a6db3831a6410983c766d382e4112ff23ec0e64ea" exitCode=0 Oct 30 00:08:55.293400 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:55.293287 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x4tph" event={"ID":"712c67e6-6358-4c10-b540-f6f66d889fb0","Type":"ContainerDied","Data":"39632d04299a346a63414f5a6db3831a6410983c766d382e4112ff23ec0e64ea"} Oct 30 00:08:58.469472 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:58.469394 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4"] Oct 30 00:08:58.511711 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:58.511356 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4"] Oct 30 00:08:58.511711 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:58.511546 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4" Oct 30 00:08:58.600288 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:58.599461 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 00:08:58.600288 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:58.599563 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 00:08:58.600288 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:58.599691 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 00:08:58.600288 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:58.599783 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 00:08:58.600288 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:58.600229 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 00:08:58.601535 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:58.601488 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-84fdv\"" Oct 30 00:08:58.637798 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:58.637755 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d42305a4-b364-4f61-b321-ea9c0557718d-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4\" (UID: \"d42305a4-b364-4f61-b321-ea9c0557718d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4" Oct 30 00:08:58.638004 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:58.637805 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d42305a4-b364-4f61-b321-ea9c0557718d-tmp\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4\" (UID: \"d42305a4-b364-4f61-b321-ea9c0557718d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4" Oct 30 00:08:58.638004 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:58.637856 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d42305a4-b364-4f61-b321-ea9c0557718d-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4\" (UID: \"d42305a4-b364-4f61-b321-ea9c0557718d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4" Oct 30 00:08:58.638004 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:58.637897 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d42305a4-b364-4f61-b321-ea9c0557718d-manifests\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4\" (UID: \"d42305a4-b364-4f61-b321-ea9c0557718d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4" Oct 30 00:08:58.638004 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:58.637939 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t27w2\" (UniqueName: \"kubernetes.io/projected/d42305a4-b364-4f61-b321-ea9c0557718d-kube-api-access-t27w2\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4\" (UID: \"d42305a4-b364-4f61-b321-ea9c0557718d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4" Oct 30 00:08:58.638004 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:58.637992 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d42305a4-b364-4f61-b321-ea9c0557718d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4\" (UID: \"d42305a4-b364-4f61-b321-ea9c0557718d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4" Oct 30 00:08:58.739741 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:58.738896 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d42305a4-b364-4f61-b321-ea9c0557718d-manifests\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4\" (UID: \"d42305a4-b364-4f61-b321-ea9c0557718d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4" Oct 30 00:08:58.739741 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:58.738970 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t27w2\" (UniqueName: \"kubernetes.io/projected/d42305a4-b364-4f61-b321-ea9c0557718d-kube-api-access-t27w2\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4\" (UID: \"d42305a4-b364-4f61-b321-ea9c0557718d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4" Oct 30 00:08:58.739741 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:58.739021 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d42305a4-b364-4f61-b321-ea9c0557718d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4\" (UID: \"d42305a4-b364-4f61-b321-ea9c0557718d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4" Oct 30 00:08:58.739741 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:58.739137 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d42305a4-b364-4f61-b321-ea9c0557718d-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4\" (UID: \"d42305a4-b364-4f61-b321-ea9c0557718d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4" Oct 30 00:08:58.739741 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:58.739161 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d42305a4-b364-4f61-b321-ea9c0557718d-tmp\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4\" (UID: \"d42305a4-b364-4f61-b321-ea9c0557718d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4" Oct 30 00:08:58.739741 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:58.739207 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d42305a4-b364-4f61-b321-ea9c0557718d-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4\" (UID: \"d42305a4-b364-4f61-b321-ea9c0557718d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4" Oct 30 00:08:58.742728 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:58.742370 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d42305a4-b364-4f61-b321-ea9c0557718d-manifests\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4\" (UID: \"d42305a4-b364-4f61-b321-ea9c0557718d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4" Oct 30 00:08:58.747355 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:58.747257 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d42305a4-b364-4f61-b321-ea9c0557718d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4\" (UID: \"d42305a4-b364-4f61-b321-ea9c0557718d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4" Oct 30 00:08:58.748109 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:58.748057 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d42305a4-b364-4f61-b321-ea9c0557718d-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4\" (UID: \"d42305a4-b364-4f61-b321-ea9c0557718d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4" Oct 30 00:08:58.751364 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:58.751275 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d42305a4-b364-4f61-b321-ea9c0557718d-tmp\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4\" (UID: \"d42305a4-b364-4f61-b321-ea9c0557718d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4" Oct 30 00:08:58.761332 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:58.761283 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d42305a4-b364-4f61-b321-ea9c0557718d-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4\" (UID: \"d42305a4-b364-4f61-b321-ea9c0557718d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4" Oct 30 00:08:58.785989 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:58.785863 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t27w2\" (UniqueName: \"kubernetes.io/projected/d42305a4-b364-4f61-b321-ea9c0557718d-kube-api-access-t27w2\") pod \"opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4\" (UID: \"d42305a4-b364-4f61-b321-ea9c0557718d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4" Oct 30 00:08:58.912116 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:58.909787 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4" Oct 30 00:08:59.747943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:08:59.747845 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4"] Oct 30 00:09:00.364310 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:00.364199 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4" event={"ID":"d42305a4-b364-4f61-b321-ea9c0557718d","Type":"ContainerStarted","Data":"02babe29db76b0e2aeef2f30c16bcfc3518a5bc990d49e52efb9c94097adc5da"} Oct 30 00:09:00.371054 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:00.370053 5001 generic.go:356] "Generic (PLEG): container finished" podID="9c977822-ba67-47f8-9a59-fd265de65217" containerID="23ef25fa9575fdf711b5be8f04fb4545952fbfca22a52566ea5f76517786966c" exitCode=0 Oct 30 00:09:00.371054 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:00.370130 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8c8zc" event={"ID":"9c977822-ba67-47f8-9a59-fd265de65217","Type":"ContainerDied","Data":"23ef25fa9575fdf711b5be8f04fb4545952fbfca22a52566ea5f76517786966c"} Oct 30 00:09:01.397816 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:01.397776 5001 generic.go:356] "Generic (PLEG): container finished" podID="b44f71d9-0521-4d58-a1a4-00aa0b0b11f2" containerID="ed52c0ea549322a9986135a0ab256f3cb66e50294ffca593d464319be53d8756" exitCode=0 Oct 30 00:09:01.397816 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:01.397896 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449hnf2" event={"ID":"b44f71d9-0521-4d58-a1a4-00aa0b0b11f2","Type":"ContainerDied","Data":"ed52c0ea549322a9986135a0ab256f3cb66e50294ffca593d464319be53d8756"} Oct 30 00:09:10.561426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:10.560789 5001 generic.go:356] "Generic (PLEG): container finished" podID="712c67e6-6358-4c10-b540-f6f66d889fb0" containerID="b27ae8717a9e27e7fbb6082fcdfb0887983b90b88b84e2b20a18cb547fd14e17" exitCode=0 Oct 30 00:09:10.561426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:10.560948 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x4tph" event={"ID":"712c67e6-6358-4c10-b540-f6f66d889fb0","Type":"ContainerDied","Data":"b27ae8717a9e27e7fbb6082fcdfb0887983b90b88b84e2b20a18cb547fd14e17"} Oct 30 00:09:10.570255 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:10.569986 5001 generic.go:356] "Generic (PLEG): container finished" podID="4484ecc5-88d4-4f8e-8244-a7b35f95bb3a" containerID="f15a18fb14a0851a4380ecc4b2e8e295505f13bc04e215ef1289336944b2d464" exitCode=0 Oct 30 00:09:10.570255 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:10.570045 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716pmws" event={"ID":"4484ecc5-88d4-4f8e-8244-a7b35f95bb3a","Type":"ContainerDied","Data":"f15a18fb14a0851a4380ecc4b2e8e295505f13bc04e215ef1289336944b2d464"} Oct 30 00:09:11.940589 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:11.940130 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x4tph" Oct 30 00:09:11.961442 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:11.961411 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716pmws" Oct 30 00:09:12.086958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:12.086927 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4484ecc5-88d4-4f8e-8244-a7b35f95bb3a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4484ecc5-88d4-4f8e-8244-a7b35f95bb3a-bundle\") pod \"4484ecc5-88d4-4f8e-8244-a7b35f95bb3a\" (UID: \"4484ecc5-88d4-4f8e-8244-a7b35f95bb3a\") " Oct 30 00:09:12.087120 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:12.086979 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4484ecc5-88d4-4f8e-8244-a7b35f95bb3a-kube-api-access-ldnhb\" (UniqueName: \"kubernetes.io/projected/4484ecc5-88d4-4f8e-8244-a7b35f95bb3a-kube-api-access-ldnhb\") pod \"4484ecc5-88d4-4f8e-8244-a7b35f95bb3a\" (UID: \"4484ecc5-88d4-4f8e-8244-a7b35f95bb3a\") " Oct 30 00:09:12.087120 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:12.087002 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/712c67e6-6358-4c10-b540-f6f66d889fb0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/712c67e6-6358-4c10-b540-f6f66d889fb0-bundle\") pod \"712c67e6-6358-4c10-b540-f6f66d889fb0\" (UID: \"712c67e6-6358-4c10-b540-f6f66d889fb0\") " Oct 30 00:09:12.087120 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:12.087024 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4484ecc5-88d4-4f8e-8244-a7b35f95bb3a-util\" (UniqueName: \"kubernetes.io/empty-dir/4484ecc5-88d4-4f8e-8244-a7b35f95bb3a-util\") pod \"4484ecc5-88d4-4f8e-8244-a7b35f95bb3a\" (UID: \"4484ecc5-88d4-4f8e-8244-a7b35f95bb3a\") " Oct 30 00:09:12.087120 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:12.087065 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/712c67e6-6358-4c10-b540-f6f66d889fb0-util\" (UniqueName: \"kubernetes.io/empty-dir/712c67e6-6358-4c10-b540-f6f66d889fb0-util\") pod \"712c67e6-6358-4c10-b540-f6f66d889fb0\" (UID: \"712c67e6-6358-4c10-b540-f6f66d889fb0\") " Oct 30 00:09:12.087120 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:12.087081 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/712c67e6-6358-4c10-b540-f6f66d889fb0-kube-api-access-9cmm7\" (UniqueName: \"kubernetes.io/projected/712c67e6-6358-4c10-b540-f6f66d889fb0-kube-api-access-9cmm7\") pod \"712c67e6-6358-4c10-b540-f6f66d889fb0\" (UID: \"712c67e6-6358-4c10-b540-f6f66d889fb0\") " Oct 30 00:09:12.088557 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:12.088502 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4484ecc5-88d4-4f8e-8244-a7b35f95bb3a-bundle" pod "4484ecc5-88d4-4f8e-8244-a7b35f95bb3a" (UID: "4484ecc5-88d4-4f8e-8244-a7b35f95bb3a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:09:12.089278 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:12.089248 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/712c67e6-6358-4c10-b540-f6f66d889fb0-bundle" pod "712c67e6-6358-4c10-b540-f6f66d889fb0" (UID: "712c67e6-6358-4c10-b540-f6f66d889fb0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:09:12.095337 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:12.095312 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4484ecc5-88d4-4f8e-8244-a7b35f95bb3a-util" pod "4484ecc5-88d4-4f8e-8244-a7b35f95bb3a" (UID: "4484ecc5-88d4-4f8e-8244-a7b35f95bb3a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:09:12.097496 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:12.097438 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/712c67e6-6358-4c10-b540-f6f66d889fb0-util" pod "712c67e6-6358-4c10-b540-f6f66d889fb0" (UID: "712c67e6-6358-4c10-b540-f6f66d889fb0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:09:12.110865 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:12.110818 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/712c67e6-6358-4c10-b540-f6f66d889fb0-kube-api-access-9cmm7" pod "712c67e6-6358-4c10-b540-f6f66d889fb0" (UID: "712c67e6-6358-4c10-b540-f6f66d889fb0"). InnerVolumeSpecName "kube-api-access-9cmm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:09:12.113888 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:12.113847 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4484ecc5-88d4-4f8e-8244-a7b35f95bb3a-kube-api-access-ldnhb" pod "4484ecc5-88d4-4f8e-8244-a7b35f95bb3a" (UID: "4484ecc5-88d4-4f8e-8244-a7b35f95bb3a"). InnerVolumeSpecName "kube-api-access-ldnhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:09:12.188383 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:12.188298 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldnhb\" (UniqueName: \"kubernetes.io/projected/4484ecc5-88d4-4f8e-8244-a7b35f95bb3a-kube-api-access-ldnhb\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:09:12.188383 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:12.188343 5001 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/712c67e6-6358-4c10-b540-f6f66d889fb0-bundle\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:09:12.188383 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:12.188358 5001 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4484ecc5-88d4-4f8e-8244-a7b35f95bb3a-util\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:09:12.188383 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:12.188374 5001 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/712c67e6-6358-4c10-b540-f6f66d889fb0-util\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:09:12.188383 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:12.188388 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cmm7\" (UniqueName: \"kubernetes.io/projected/712c67e6-6358-4c10-b540-f6f66d889fb0-kube-api-access-9cmm7\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:09:12.188847 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:12.188401 5001 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4484ecc5-88d4-4f8e-8244-a7b35f95bb3a-bundle\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:09:12.599967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:12.596973 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x4tph" event={"ID":"712c67e6-6358-4c10-b540-f6f66d889fb0","Type":"ContainerDied","Data":"b89c4af907f787da9f4531c4ec5b520b47bc28edf1d871b381425582b8f6f5b4"} Oct 30 00:09:12.599967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:12.597035 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b89c4af907f787da9f4531c4ec5b520b47bc28edf1d871b381425582b8f6f5b4" Oct 30 00:09:12.599967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:12.597138 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x4tph" Oct 30 00:09:12.608136 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:12.604297 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716pmws" event={"ID":"4484ecc5-88d4-4f8e-8244-a7b35f95bb3a","Type":"ContainerDied","Data":"d4e7eaeb08f3a895fd759fb62b18e9ad823e2044e2dba5c0a20da19afe7a4b96"} Oct 30 00:09:12.608136 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:12.604348 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4e7eaeb08f3a895fd759fb62b18e9ad823e2044e2dba5c0a20da19afe7a4b96" Oct 30 00:09:12.608136 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:12.604422 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716pmws" Oct 30 00:09:12.616528 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:12.616154 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4" event={"ID":"d42305a4-b364-4f61-b321-ea9c0557718d","Type":"ContainerStarted","Data":"a5c2d52cefdd66c18b5f5fcb9a3e8746e002b1bee14fa69090b01e2c1d7f4047"} Oct 30 00:09:13.645386 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:13.645316 5001 generic.go:356] "Generic (PLEG): container finished" podID="d42305a4-b364-4f61-b321-ea9c0557718d" containerID="a5c2d52cefdd66c18b5f5fcb9a3e8746e002b1bee14fa69090b01e2c1d7f4047" exitCode=0 Oct 30 00:09:13.645386 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:13.645388 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4" event={"ID":"d42305a4-b364-4f61-b321-ea9c0557718d","Type":"ContainerDied","Data":"a5c2d52cefdd66c18b5f5fcb9a3e8746e002b1bee14fa69090b01e2c1d7f4047"} Oct 30 00:09:14.656571 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:14.656526 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4" event={"ID":"d42305a4-b364-4f61-b321-ea9c0557718d","Type":"ContainerStarted","Data":"24d74ca50a78d9ba9755b6140eddd0360623ae4b3a81ba10590c490cc503657d"} Oct 30 00:09:14.657067 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:14.656733 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4" Oct 30 00:09:14.734174 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:14.734115 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4" podStartSLOduration=3.297408888 podStartE2EDuration="16.734097118s" totalImagesPullingTime="12.76717125s" totalInitContainerRuntime="669.51698ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:08:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:09:14.722605789 +0000 UTC m=+269.928106969" watchObservedRunningTime="2026-10-30 00:09:14.734097118 +0000 UTC m=+269.939598299" Oct 30 00:09:15.675624 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:15.675581 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8c8zc" event={"ID":"9c977822-ba67-47f8-9a59-fd265de65217","Type":"ContainerStarted","Data":"5ef41d6c583ac574b1066dac05fdd88c2d18819d743d282c81d059e9763f53b9"} Oct 30 00:09:15.681237 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:15.681207 5001 generic.go:356] "Generic (PLEG): container finished" podID="5d34d892-f9ab-4bd2-920c-2c7305c7c57b" containerID="31e168bd51da3186175188ebab35dc9ffa609c88058a36bca2bbd0abd0cdeaaa" exitCode=0 Oct 30 00:09:15.681354 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:15.681320 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c25w57" event={"ID":"5d34d892-f9ab-4bd2-920c-2c7305c7c57b","Type":"ContainerDied","Data":"31e168bd51da3186175188ebab35dc9ffa609c88058a36bca2bbd0abd0cdeaaa"} Oct 30 00:09:15.684524 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:15.684503 5001 generic.go:356] "Generic (PLEG): container finished" podID="b44f71d9-0521-4d58-a1a4-00aa0b0b11f2" containerID="49bfe684e9cdbdb5bb8306a6602619fad80a03e2404c46fef958b106d73a7230" exitCode=0 Oct 30 00:09:15.685291 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:15.685263 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449hnf2" event={"ID":"b44f71d9-0521-4d58-a1a4-00aa0b0b11f2","Type":"ContainerDied","Data":"49bfe684e9cdbdb5bb8306a6602619fad80a03e2404c46fef958b106d73a7230"} Oct 30 00:09:15.717515 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:15.717453 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8c8zc" podStartSLOduration=3.747224591 podStartE2EDuration="44.717432979s" totalImagesPullingTime="40.970208388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:08:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:09:15.714737967 +0000 UTC m=+270.920239149" watchObservedRunningTime="2026-10-30 00:09:15.717432979 +0000 UTC m=+270.922934161" Oct 30 00:09:16.691391 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:16.691360 5001 generic.go:356] "Generic (PLEG): container finished" podID="9c977822-ba67-47f8-9a59-fd265de65217" containerID="5ef41d6c583ac574b1066dac05fdd88c2d18819d743d282c81d059e9763f53b9" exitCode=0 Oct 30 00:09:16.691846 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:16.691435 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8c8zc" event={"ID":"9c977822-ba67-47f8-9a59-fd265de65217","Type":"ContainerDied","Data":"5ef41d6c583ac574b1066dac05fdd88c2d18819d743d282c81d059e9763f53b9"} Oct 30 00:09:16.693607 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:16.693575 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atzn5p" event={"ID":"bbffa049-445d-47f2-bcf6-6edd14cf25c1","Type":"ContainerStarted","Data":"8508525243c8a50254c8cdb271c2a483cf7c8c6b35fedd664a133118ecd551b1"} Oct 30 00:09:16.695796 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:16.695776 5001 generic.go:356] "Generic (PLEG): container finished" podID="08d02afc-16dd-4010-9292-4d394739a984" containerID="2c6567713eff3068c0825da05ab99ab646ffc16e207e2edced040a0b7253dfdf" exitCode=0 Oct 30 00:09:16.695884 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:16.695845 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5w4t2" event={"ID":"08d02afc-16dd-4010-9292-4d394739a984","Type":"ContainerDied","Data":"2c6567713eff3068c0825da05ab99ab646ffc16e207e2edced040a0b7253dfdf"} Oct 30 00:09:16.697892 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:16.697871 5001 generic.go:356] "Generic (PLEG): container finished" podID="7d1bbe05-289d-414a-8854-db469519e37e" containerID="e85284973b9d19f48d16ba311f12690d1105e024f992ea7b475fb6a65026ffd1" exitCode=0 Oct 30 00:09:16.698007 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:16.697938 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c665gj4" event={"ID":"7d1bbe05-289d-414a-8854-db469519e37e","Type":"ContainerDied","Data":"e85284973b9d19f48d16ba311f12690d1105e024f992ea7b475fb6a65026ffd1"} Oct 30 00:09:16.872133 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:16.872104 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c25w57" Oct 30 00:09:16.877004 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:16.876960 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atzn5p" podStartSLOduration=3.111538706 podStartE2EDuration="38.876945856s" totalImagesPullingTime="35.76540715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:08:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:09:16.875387501 +0000 UTC m=+272.080888698" watchObservedRunningTime="2026-10-30 00:09:16.876945856 +0000 UTC m=+272.082447046" Oct 30 00:09:16.891492 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:16.891466 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449hnf2" Oct 30 00:09:16.940727 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:16.940698 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d34d892-f9ab-4bd2-920c-2c7305c7c57b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d34d892-f9ab-4bd2-920c-2c7305c7c57b-bundle\") pod \"5d34d892-f9ab-4bd2-920c-2c7305c7c57b\" (UID: \"5d34d892-f9ab-4bd2-920c-2c7305c7c57b\") " Oct 30 00:09:16.941039 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:16.940748 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b44f71d9-0521-4d58-a1a4-00aa0b0b11f2-kube-api-access-mjwfn\" (UniqueName: \"kubernetes.io/projected/b44f71d9-0521-4d58-a1a4-00aa0b0b11f2-kube-api-access-mjwfn\") pod \"b44f71d9-0521-4d58-a1a4-00aa0b0b11f2\" (UID: \"b44f71d9-0521-4d58-a1a4-00aa0b0b11f2\") " Oct 30 00:09:16.941039 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:16.940781 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b44f71d9-0521-4d58-a1a4-00aa0b0b11f2-util\" (UniqueName: \"kubernetes.io/empty-dir/b44f71d9-0521-4d58-a1a4-00aa0b0b11f2-util\") pod \"b44f71d9-0521-4d58-a1a4-00aa0b0b11f2\" (UID: \"b44f71d9-0521-4d58-a1a4-00aa0b0b11f2\") " Oct 30 00:09:16.941039 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:16.940806 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d34d892-f9ab-4bd2-920c-2c7305c7c57b-util\" (UniqueName: \"kubernetes.io/empty-dir/5d34d892-f9ab-4bd2-920c-2c7305c7c57b-util\") pod \"5d34d892-f9ab-4bd2-920c-2c7305c7c57b\" (UID: \"5d34d892-f9ab-4bd2-920c-2c7305c7c57b\") " Oct 30 00:09:16.941039 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:16.940828 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b44f71d9-0521-4d58-a1a4-00aa0b0b11f2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b44f71d9-0521-4d58-a1a4-00aa0b0b11f2-bundle\") pod \"b44f71d9-0521-4d58-a1a4-00aa0b0b11f2\" (UID: \"b44f71d9-0521-4d58-a1a4-00aa0b0b11f2\") " Oct 30 00:09:16.941039 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:16.940893 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d34d892-f9ab-4bd2-920c-2c7305c7c57b-kube-api-access-l768r\" (UniqueName: \"kubernetes.io/projected/5d34d892-f9ab-4bd2-920c-2c7305c7c57b-kube-api-access-l768r\") pod \"5d34d892-f9ab-4bd2-920c-2c7305c7c57b\" (UID: \"5d34d892-f9ab-4bd2-920c-2c7305c7c57b\") " Oct 30 00:09:16.941381 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:16.941351 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d34d892-f9ab-4bd2-920c-2c7305c7c57b-bundle" pod "5d34d892-f9ab-4bd2-920c-2c7305c7c57b" (UID: "5d34d892-f9ab-4bd2-920c-2c7305c7c57b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:09:16.941788 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:16.941643 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44f71d9-0521-4d58-a1a4-00aa0b0b11f2-bundle" pod "b44f71d9-0521-4d58-a1a4-00aa0b0b11f2" (UID: "b44f71d9-0521-4d58-a1a4-00aa0b0b11f2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:09:16.943387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:16.943364 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d34d892-f9ab-4bd2-920c-2c7305c7c57b-kube-api-access-l768r" pod "5d34d892-f9ab-4bd2-920c-2c7305c7c57b" (UID: "5d34d892-f9ab-4bd2-920c-2c7305c7c57b"). InnerVolumeSpecName "kube-api-access-l768r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:09:16.943545 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:16.943526 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44f71d9-0521-4d58-a1a4-00aa0b0b11f2-kube-api-access-mjwfn" pod "b44f71d9-0521-4d58-a1a4-00aa0b0b11f2" (UID: "b44f71d9-0521-4d58-a1a4-00aa0b0b11f2"). InnerVolumeSpecName "kube-api-access-mjwfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:09:16.946766 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:16.946745 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44f71d9-0521-4d58-a1a4-00aa0b0b11f2-util" pod "b44f71d9-0521-4d58-a1a4-00aa0b0b11f2" (UID: "b44f71d9-0521-4d58-a1a4-00aa0b0b11f2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:09:16.947603 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:16.947582 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d34d892-f9ab-4bd2-920c-2c7305c7c57b-util" pod "5d34d892-f9ab-4bd2-920c-2c7305c7c57b" (UID: "5d34d892-f9ab-4bd2-920c-2c7305c7c57b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:09:17.042617 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.042581 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjwfn\" (UniqueName: \"kubernetes.io/projected/b44f71d9-0521-4d58-a1a4-00aa0b0b11f2-kube-api-access-mjwfn\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:09:17.042617 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.042611 5001 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b44f71d9-0521-4d58-a1a4-00aa0b0b11f2-util\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:09:17.042617 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.042623 5001 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d34d892-f9ab-4bd2-920c-2c7305c7c57b-util\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:09:17.042836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.042631 5001 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b44f71d9-0521-4d58-a1a4-00aa0b0b11f2-bundle\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:09:17.042836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.042639 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l768r\" (UniqueName: \"kubernetes.io/projected/5d34d892-f9ab-4bd2-920c-2c7305c7c57b-kube-api-access-l768r\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:09:17.042836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.042647 5001 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d34d892-f9ab-4bd2-920c-2c7305c7c57b-bundle\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:09:17.321759 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:09:17.321669 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:09:17.704968 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.704864 5001 generic.go:356] "Generic (PLEG): container finished" podID="556143f7-0f4b-456c-98c1-da699bc3c9c3" containerID="efe456445d922e10a99851805c30b8ff9157f2cb536e439b616024ff29341ba0" exitCode=0 Oct 30 00:09:17.705353 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.704957 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pvbrf" event={"ID":"556143f7-0f4b-456c-98c1-da699bc3c9c3","Type":"ContainerDied","Data":"efe456445d922e10a99851805c30b8ff9157f2cb536e439b616024ff29341ba0"} Oct 30 00:09:17.707011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.706984 5001 generic.go:356] "Generic (PLEG): container finished" podID="bbffa049-445d-47f2-bcf6-6edd14cf25c1" containerID="8508525243c8a50254c8cdb271c2a483cf7c8c6b35fedd664a133118ecd551b1" exitCode=0 Oct 30 00:09:17.707149 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.707008 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atzn5p" event={"ID":"bbffa049-445d-47f2-bcf6-6edd14cf25c1","Type":"ContainerDied","Data":"8508525243c8a50254c8cdb271c2a483cf7c8c6b35fedd664a133118ecd551b1"} Oct 30 00:09:17.709052 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.709032 5001 generic.go:356] "Generic (PLEG): container finished" podID="95276631-3e7c-4d4b-a182-8b5d0fa319b8" containerID="e4321c2f7e1dc2ef535d3bc81e9098c440d24a6dd77fbd2a6cd11ad90a404427" exitCode=0 Oct 30 00:09:17.709165 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.709054 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxvr2f" event={"ID":"95276631-3e7c-4d4b-a182-8b5d0fa319b8","Type":"ContainerDied","Data":"e4321c2f7e1dc2ef535d3bc81e9098c440d24a6dd77fbd2a6cd11ad90a404427"} Oct 30 00:09:17.710756 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.710741 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c25w57" Oct 30 00:09:17.710853 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.710756 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c25w57" event={"ID":"5d34d892-f9ab-4bd2-920c-2c7305c7c57b","Type":"ContainerDied","Data":"745af6a21c1e0250ce0199077d629269e76b4d250668ca5aed823e654e57b1ae"} Oct 30 00:09:17.710853 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.710792 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="745af6a21c1e0250ce0199077d629269e76b4d250668ca5aed823e654e57b1ae" Oct 30 00:09:17.712617 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.712594 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449hnf2" Oct 30 00:09:17.712617 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.712607 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449hnf2" event={"ID":"b44f71d9-0521-4d58-a1a4-00aa0b0b11f2","Type":"ContainerDied","Data":"c18ee7babcb4149baa27757a1d376939947c022674db5e445d00a4711d155d43"} Oct 30 00:09:17.712764 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.712635 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c18ee7babcb4149baa27757a1d376939947c022674db5e445d00a4711d155d43" Oct 30 00:09:17.714859 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.714841 5001 generic.go:356] "Generic (PLEG): container finished" podID="a78c8440-3039-46ab-b3c6-d0276d984f56" containerID="e4e8a88af5a16dd1e5f2a62611c88043a67064209801e088026b933fd72911ae" exitCode=0 Oct 30 00:09:17.714976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.714953 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dlg25" event={"ID":"a78c8440-3039-46ab-b3c6-d0276d984f56","Type":"ContainerDied","Data":"e4e8a88af5a16dd1e5f2a62611c88043a67064209801e088026b933fd72911ae"} Oct 30 00:09:17.862997 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.862967 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5w4t2" Oct 30 00:09:17.897665 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.897642 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8c8zc" Oct 30 00:09:17.900543 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.900525 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c665gj4" Oct 30 00:09:17.950239 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.950199 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c977822-ba67-47f8-9a59-fd265de65217-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c977822-ba67-47f8-9a59-fd265de65217-bundle\") pod \"9c977822-ba67-47f8-9a59-fd265de65217\" (UID: \"9c977822-ba67-47f8-9a59-fd265de65217\") " Oct 30 00:09:17.950239 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.950244 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c977822-ba67-47f8-9a59-fd265de65217-util\" (UniqueName: \"kubernetes.io/empty-dir/9c977822-ba67-47f8-9a59-fd265de65217-util\") pod \"9c977822-ba67-47f8-9a59-fd265de65217\" (UID: \"9c977822-ba67-47f8-9a59-fd265de65217\") " Oct 30 00:09:17.950505 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.950272 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d1bbe05-289d-414a-8854-db469519e37e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d1bbe05-289d-414a-8854-db469519e37e-bundle\") pod \"7d1bbe05-289d-414a-8854-db469519e37e\" (UID: \"7d1bbe05-289d-414a-8854-db469519e37e\") " Oct 30 00:09:17.950505 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.950316 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d1bbe05-289d-414a-8854-db469519e37e-kube-api-access-7fkbq\" (UniqueName: \"kubernetes.io/projected/7d1bbe05-289d-414a-8854-db469519e37e-kube-api-access-7fkbq\") pod \"7d1bbe05-289d-414a-8854-db469519e37e\" (UID: \"7d1bbe05-289d-414a-8854-db469519e37e\") " Oct 30 00:09:17.950505 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.950342 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08d02afc-16dd-4010-9292-4d394739a984-bundle\" (UniqueName: \"kubernetes.io/empty-dir/08d02afc-16dd-4010-9292-4d394739a984-bundle\") pod \"08d02afc-16dd-4010-9292-4d394739a984\" (UID: \"08d02afc-16dd-4010-9292-4d394739a984\") " Oct 30 00:09:17.950505 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.950380 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08d02afc-16dd-4010-9292-4d394739a984-util\" (UniqueName: \"kubernetes.io/empty-dir/08d02afc-16dd-4010-9292-4d394739a984-util\") pod \"08d02afc-16dd-4010-9292-4d394739a984\" (UID: \"08d02afc-16dd-4010-9292-4d394739a984\") " Oct 30 00:09:17.950505 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.950433 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d1bbe05-289d-414a-8854-db469519e37e-util\" (UniqueName: \"kubernetes.io/empty-dir/7d1bbe05-289d-414a-8854-db469519e37e-util\") pod \"7d1bbe05-289d-414a-8854-db469519e37e\" (UID: \"7d1bbe05-289d-414a-8854-db469519e37e\") " Oct 30 00:09:17.950505 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.950482 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c977822-ba67-47f8-9a59-fd265de65217-kube-api-access-qn4nb\" (UniqueName: \"kubernetes.io/projected/9c977822-ba67-47f8-9a59-fd265de65217-kube-api-access-qn4nb\") pod \"9c977822-ba67-47f8-9a59-fd265de65217\" (UID: \"9c977822-ba67-47f8-9a59-fd265de65217\") " Oct 30 00:09:17.950806 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.950525 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08d02afc-16dd-4010-9292-4d394739a984-kube-api-access-cbh6p\" (UniqueName: \"kubernetes.io/projected/08d02afc-16dd-4010-9292-4d394739a984-kube-api-access-cbh6p\") pod \"08d02afc-16dd-4010-9292-4d394739a984\" (UID: \"08d02afc-16dd-4010-9292-4d394739a984\") " Oct 30 00:09:17.950979 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.950950 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1bbe05-289d-414a-8854-db469519e37e-bundle" pod "7d1bbe05-289d-414a-8854-db469519e37e" (UID: "7d1bbe05-289d-414a-8854-db469519e37e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:09:17.951121 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.951095 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c977822-ba67-47f8-9a59-fd265de65217-bundle" pod "9c977822-ba67-47f8-9a59-fd265de65217" (UID: "9c977822-ba67-47f8-9a59-fd265de65217"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:09:17.952215 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.952186 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08d02afc-16dd-4010-9292-4d394739a984-bundle" pod "08d02afc-16dd-4010-9292-4d394739a984" (UID: "08d02afc-16dd-4010-9292-4d394739a984"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:09:17.953055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.953030 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d1bbe05-289d-414a-8854-db469519e37e-kube-api-access-7fkbq" pod "7d1bbe05-289d-414a-8854-db469519e37e" (UID: "7d1bbe05-289d-414a-8854-db469519e37e"). InnerVolumeSpecName "kube-api-access-7fkbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:09:17.953416 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.953385 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08d02afc-16dd-4010-9292-4d394739a984-kube-api-access-cbh6p" pod "08d02afc-16dd-4010-9292-4d394739a984" (UID: "08d02afc-16dd-4010-9292-4d394739a984"). InnerVolumeSpecName "kube-api-access-cbh6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:09:17.953566 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.953548 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c977822-ba67-47f8-9a59-fd265de65217-kube-api-access-qn4nb" pod "9c977822-ba67-47f8-9a59-fd265de65217" (UID: "9c977822-ba67-47f8-9a59-fd265de65217"). InnerVolumeSpecName "kube-api-access-qn4nb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:09:17.958386 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.958313 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1bbe05-289d-414a-8854-db469519e37e-util" pod "7d1bbe05-289d-414a-8854-db469519e37e" (UID: "7d1bbe05-289d-414a-8854-db469519e37e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:09:17.958763 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.958721 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08d02afc-16dd-4010-9292-4d394739a984-util" pod "08d02afc-16dd-4010-9292-4d394739a984" (UID: "08d02afc-16dd-4010-9292-4d394739a984"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:09:17.958994 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:17.958968 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c977822-ba67-47f8-9a59-fd265de65217-util" pod "9c977822-ba67-47f8-9a59-fd265de65217" (UID: "9c977822-ba67-47f8-9a59-fd265de65217"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:09:18.051595 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.051562 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn4nb\" (UniqueName: \"kubernetes.io/projected/9c977822-ba67-47f8-9a59-fd265de65217-kube-api-access-qn4nb\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:09:18.051595 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.051591 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbh6p\" (UniqueName: \"kubernetes.io/projected/08d02afc-16dd-4010-9292-4d394739a984-kube-api-access-cbh6p\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:09:18.051595 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.051601 5001 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c977822-ba67-47f8-9a59-fd265de65217-bundle\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:09:18.051825 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.051612 5001 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c977822-ba67-47f8-9a59-fd265de65217-util\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:09:18.051825 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.051621 5001 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d1bbe05-289d-414a-8854-db469519e37e-bundle\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:09:18.051825 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.051631 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fkbq\" (UniqueName: \"kubernetes.io/projected/7d1bbe05-289d-414a-8854-db469519e37e-kube-api-access-7fkbq\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:09:18.051825 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.051640 5001 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08d02afc-16dd-4010-9292-4d394739a984-bundle\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:09:18.051825 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.051648 5001 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08d02afc-16dd-4010-9292-4d394739a984-util\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:09:18.051825 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.051657 5001 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d1bbe05-289d-414a-8854-db469519e37e-util\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:09:18.721645 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.721616 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8c8zc" Oct 30 00:09:18.722077 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.721646 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8c8zc" event={"ID":"9c977822-ba67-47f8-9a59-fd265de65217","Type":"ContainerDied","Data":"ec96b29fcec58f8a9048b190592732c91c8b934d44877d6332319a6e47c47811"} Oct 30 00:09:18.722077 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.721695 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec96b29fcec58f8a9048b190592732c91c8b934d44877d6332319a6e47c47811" Oct 30 00:09:18.723766 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.723746 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5w4t2" Oct 30 00:09:18.723900 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.723741 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5w4t2" event={"ID":"08d02afc-16dd-4010-9292-4d394739a984","Type":"ContainerDied","Data":"5361489d895dfcacd14153693a8e556a86734867a5cc42350a298e9c76c8aeb3"} Oct 30 00:09:18.723900 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.723865 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5361489d895dfcacd14153693a8e556a86734867a5cc42350a298e9c76c8aeb3" Oct 30 00:09:18.725704 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.725685 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c665gj4" Oct 30 00:09:18.725797 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.725726 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c665gj4" event={"ID":"7d1bbe05-289d-414a-8854-db469519e37e","Type":"ContainerDied","Data":"9b9b60f52b477fff618468d22a9b5eb3d01c7ebcf9e982e1bef91a39dc022518"} Oct 30 00:09:18.725797 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.725768 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b9b60f52b477fff618468d22a9b5eb3d01c7ebcf9e982e1bef91a39dc022518" Oct 30 00:09:18.877472 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.877443 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atzn5p" Oct 30 00:09:18.927627 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.927600 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dlg25" Oct 30 00:09:18.934641 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.934614 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxvr2f" Oct 30 00:09:18.942096 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.942078 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pvbrf" Oct 30 00:09:18.960222 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.960194 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a78c8440-3039-46ab-b3c6-d0276d984f56-util\" (UniqueName: \"kubernetes.io/empty-dir/a78c8440-3039-46ab-b3c6-d0276d984f56-util\") pod \"a78c8440-3039-46ab-b3c6-d0276d984f56\" (UID: \"a78c8440-3039-46ab-b3c6-d0276d984f56\") " Oct 30 00:09:18.960365 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.960240 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/556143f7-0f4b-456c-98c1-da699bc3c9c3-kube-api-access-2xsjj\" (UniqueName: \"kubernetes.io/projected/556143f7-0f4b-456c-98c1-da699bc3c9c3-kube-api-access-2xsjj\") pod \"556143f7-0f4b-456c-98c1-da699bc3c9c3\" (UID: \"556143f7-0f4b-456c-98c1-da699bc3c9c3\") " Oct 30 00:09:18.960365 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.960289 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95276631-3e7c-4d4b-a182-8b5d0fa319b8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/95276631-3e7c-4d4b-a182-8b5d0fa319b8-bundle\") pod \"95276631-3e7c-4d4b-a182-8b5d0fa319b8\" (UID: \"95276631-3e7c-4d4b-a182-8b5d0fa319b8\") " Oct 30 00:09:18.960365 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.960313 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/556143f7-0f4b-456c-98c1-da699bc3c9c3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/556143f7-0f4b-456c-98c1-da699bc3c9c3-bundle\") pod \"556143f7-0f4b-456c-98c1-da699bc3c9c3\" (UID: \"556143f7-0f4b-456c-98c1-da699bc3c9c3\") " Oct 30 00:09:18.960365 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.960343 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95276631-3e7c-4d4b-a182-8b5d0fa319b8-util\" (UniqueName: \"kubernetes.io/empty-dir/95276631-3e7c-4d4b-a182-8b5d0fa319b8-util\") pod \"95276631-3e7c-4d4b-a182-8b5d0fa319b8\" (UID: \"95276631-3e7c-4d4b-a182-8b5d0fa319b8\") " Oct 30 00:09:18.960581 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.960456 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbffa049-445d-47f2-bcf6-6edd14cf25c1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbffa049-445d-47f2-bcf6-6edd14cf25c1-bundle\") pod \"bbffa049-445d-47f2-bcf6-6edd14cf25c1\" (UID: \"bbffa049-445d-47f2-bcf6-6edd14cf25c1\") " Oct 30 00:09:18.960581 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.960504 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/556143f7-0f4b-456c-98c1-da699bc3c9c3-util\" (UniqueName: \"kubernetes.io/empty-dir/556143f7-0f4b-456c-98c1-da699bc3c9c3-util\") pod \"556143f7-0f4b-456c-98c1-da699bc3c9c3\" (UID: \"556143f7-0f4b-456c-98c1-da699bc3c9c3\") " Oct 30 00:09:18.960581 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.960546 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95276631-3e7c-4d4b-a182-8b5d0fa319b8-kube-api-access-7ljtz\" (UniqueName: \"kubernetes.io/projected/95276631-3e7c-4d4b-a182-8b5d0fa319b8-kube-api-access-7ljtz\") pod \"95276631-3e7c-4d4b-a182-8b5d0fa319b8\" (UID: \"95276631-3e7c-4d4b-a182-8b5d0fa319b8\") " Oct 30 00:09:18.960581 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.960575 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbffa049-445d-47f2-bcf6-6edd14cf25c1-kube-api-access-7h7ct\" (UniqueName: \"kubernetes.io/projected/bbffa049-445d-47f2-bcf6-6edd14cf25c1-kube-api-access-7h7ct\") pod \"bbffa049-445d-47f2-bcf6-6edd14cf25c1\" (UID: \"bbffa049-445d-47f2-bcf6-6edd14cf25c1\") " Oct 30 00:09:18.960777 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.960627 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a78c8440-3039-46ab-b3c6-d0276d984f56-kube-api-access-b5fgl\" (UniqueName: \"kubernetes.io/projected/a78c8440-3039-46ab-b3c6-d0276d984f56-kube-api-access-b5fgl\") pod \"a78c8440-3039-46ab-b3c6-d0276d984f56\" (UID: \"a78c8440-3039-46ab-b3c6-d0276d984f56\") " Oct 30 00:09:18.960777 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.960658 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a78c8440-3039-46ab-b3c6-d0276d984f56-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a78c8440-3039-46ab-b3c6-d0276d984f56-bundle\") pod \"a78c8440-3039-46ab-b3c6-d0276d984f56\" (UID: \"a78c8440-3039-46ab-b3c6-d0276d984f56\") " Oct 30 00:09:18.960777 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.960690 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbffa049-445d-47f2-bcf6-6edd14cf25c1-util\" (UniqueName: \"kubernetes.io/empty-dir/bbffa049-445d-47f2-bcf6-6edd14cf25c1-util\") pod \"bbffa049-445d-47f2-bcf6-6edd14cf25c1\" (UID: \"bbffa049-445d-47f2-bcf6-6edd14cf25c1\") " Oct 30 00:09:18.961001 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.960971 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/556143f7-0f4b-456c-98c1-da699bc3c9c3-bundle" pod "556143f7-0f4b-456c-98c1-da699bc3c9c3" (UID: "556143f7-0f4b-456c-98c1-da699bc3c9c3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:09:18.961059 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.961024 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95276631-3e7c-4d4b-a182-8b5d0fa319b8-bundle" pod "95276631-3e7c-4d4b-a182-8b5d0fa319b8" (UID: "95276631-3e7c-4d4b-a182-8b5d0fa319b8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:09:18.961645 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.961613 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbffa049-445d-47f2-bcf6-6edd14cf25c1-bundle" pod "bbffa049-445d-47f2-bcf6-6edd14cf25c1" (UID: "bbffa049-445d-47f2-bcf6-6edd14cf25c1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:09:18.962963 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.962937 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/556143f7-0f4b-456c-98c1-da699bc3c9c3-kube-api-access-2xsjj" pod "556143f7-0f4b-456c-98c1-da699bc3c9c3" (UID: "556143f7-0f4b-456c-98c1-da699bc3c9c3"). InnerVolumeSpecName "kube-api-access-2xsjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:09:18.963438 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.963410 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a78c8440-3039-46ab-b3c6-d0276d984f56-bundle" pod "a78c8440-3039-46ab-b3c6-d0276d984f56" (UID: "a78c8440-3039-46ab-b3c6-d0276d984f56"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:09:18.963696 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.963662 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95276631-3e7c-4d4b-a182-8b5d0fa319b8-kube-api-access-7ljtz" pod "95276631-3e7c-4d4b-a182-8b5d0fa319b8" (UID: "95276631-3e7c-4d4b-a182-8b5d0fa319b8"). InnerVolumeSpecName "kube-api-access-7ljtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:09:18.964005 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.963976 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a78c8440-3039-46ab-b3c6-d0276d984f56-kube-api-access-b5fgl" pod "a78c8440-3039-46ab-b3c6-d0276d984f56" (UID: "a78c8440-3039-46ab-b3c6-d0276d984f56"). InnerVolumeSpecName "kube-api-access-b5fgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:09:18.964288 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.964263 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbffa049-445d-47f2-bcf6-6edd14cf25c1-kube-api-access-7h7ct" pod "bbffa049-445d-47f2-bcf6-6edd14cf25c1" (UID: "bbffa049-445d-47f2-bcf6-6edd14cf25c1"). InnerVolumeSpecName "kube-api-access-7h7ct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:09:18.968825 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.968794 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/556143f7-0f4b-456c-98c1-da699bc3c9c3-util" pod "556143f7-0f4b-456c-98c1-da699bc3c9c3" (UID: "556143f7-0f4b-456c-98c1-da699bc3c9c3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:09:18.970317 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.970288 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a78c8440-3039-46ab-b3c6-d0276d984f56-util" pod "a78c8440-3039-46ab-b3c6-d0276d984f56" (UID: "a78c8440-3039-46ab-b3c6-d0276d984f56"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:09:18.970731 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.970706 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbffa049-445d-47f2-bcf6-6edd14cf25c1-util" pod "bbffa049-445d-47f2-bcf6-6edd14cf25c1" (UID: "bbffa049-445d-47f2-bcf6-6edd14cf25c1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:09:18.970817 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:18.970734 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95276631-3e7c-4d4b-a182-8b5d0fa319b8-util" pod "95276631-3e7c-4d4b-a182-8b5d0fa319b8" (UID: "95276631-3e7c-4d4b-a182-8b5d0fa319b8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:09:19.061468 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:19.061391 5001 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a78c8440-3039-46ab-b3c6-d0276d984f56-util\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:09:19.061468 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:19.061420 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xsjj\" (UniqueName: \"kubernetes.io/projected/556143f7-0f4b-456c-98c1-da699bc3c9c3-kube-api-access-2xsjj\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:09:19.061468 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:19.061430 5001 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/95276631-3e7c-4d4b-a182-8b5d0fa319b8-bundle\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:09:19.061468 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:19.061440 5001 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/556143f7-0f4b-456c-98c1-da699bc3c9c3-bundle\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:09:19.061468 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:19.061448 5001 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/95276631-3e7c-4d4b-a182-8b5d0fa319b8-util\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:09:19.061468 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:19.061458 5001 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbffa049-445d-47f2-bcf6-6edd14cf25c1-bundle\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:09:19.061468 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:19.061465 5001 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/556143f7-0f4b-456c-98c1-da699bc3c9c3-util\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:09:19.061468 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:19.061473 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ljtz\" (UniqueName: \"kubernetes.io/projected/95276631-3e7c-4d4b-a182-8b5d0fa319b8-kube-api-access-7ljtz\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:09:19.061797 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:19.061482 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h7ct\" (UniqueName: \"kubernetes.io/projected/bbffa049-445d-47f2-bcf6-6edd14cf25c1-kube-api-access-7h7ct\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:09:19.061797 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:19.061492 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5fgl\" (UniqueName: \"kubernetes.io/projected/a78c8440-3039-46ab-b3c6-d0276d984f56-kube-api-access-b5fgl\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:09:19.061797 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:19.061502 5001 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a78c8440-3039-46ab-b3c6-d0276d984f56-bundle\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:09:19.061797 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:19.061510 5001 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbffa049-445d-47f2-bcf6-6edd14cf25c1-util\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:09:19.732385 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:19.732343 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dlg25" event={"ID":"a78c8440-3039-46ab-b3c6-d0276d984f56","Type":"ContainerDied","Data":"711e6c41c60977f6617ff1940ac6901763e10355280f2bee496d02091051a664"} Oct 30 00:09:19.732809 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:19.732395 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="711e6c41c60977f6617ff1940ac6901763e10355280f2bee496d02091051a664" Oct 30 00:09:19.732809 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:19.732358 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dlg25" Oct 30 00:09:19.736470 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:19.736452 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pvbrf" Oct 30 00:09:19.736605 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:19.736485 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pvbrf" event={"ID":"556143f7-0f4b-456c-98c1-da699bc3c9c3","Type":"ContainerDied","Data":"46235b152c7ac4079d10ab1eb19218af2501735956c32de5da08553ce2e82f23"} Oct 30 00:09:19.736605 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:19.736525 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46235b152c7ac4079d10ab1eb19218af2501735956c32de5da08553ce2e82f23" Oct 30 00:09:19.738515 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:19.738499 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atzn5p" Oct 30 00:09:19.738515 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:19.738498 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atzn5p" event={"ID":"bbffa049-445d-47f2-bcf6-6edd14cf25c1","Type":"ContainerDied","Data":"a71f5451df2846f1032b846f80285c231c907657fda1f7f6e236a2977ebac98f"} Oct 30 00:09:19.738670 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:19.738540 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a71f5451df2846f1032b846f80285c231c907657fda1f7f6e236a2977ebac98f" Oct 30 00:09:19.740580 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:19.740554 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxvr2f" event={"ID":"95276631-3e7c-4d4b-a182-8b5d0fa319b8","Type":"ContainerDied","Data":"6771d7d501bf494e8c6693753964b21f519cf0f1598b0d10524905b7b57e0a52"} Oct 30 00:09:19.740665 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:19.740599 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6771d7d501bf494e8c6693753964b21f519cf0f1598b0d10524905b7b57e0a52" Oct 30 00:09:19.740665 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:19.740571 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxvr2f" Oct 30 00:09:24.684062 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684027 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lcpns"] Oct 30 00:09:24.684518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684350 5001 cpu_manager.go:414] "containerMap: removing container" podUID="712c67e6-6358-4c10-b540-f6f66d889fb0" containerName="util" Oct 30 00:09:24.684518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684361 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="712c67e6-6358-4c10-b540-f6f66d889fb0" containerName="util" Oct 30 00:09:24.684518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684371 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b44f71d9-0521-4d58-a1a4-00aa0b0b11f2" containerName="pull" Oct 30 00:09:24.684518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684376 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b44f71d9-0521-4d58-a1a4-00aa0b0b11f2" containerName="pull" Oct 30 00:09:24.684518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684383 5001 cpu_manager.go:414] "containerMap: removing container" podUID="712c67e6-6358-4c10-b540-f6f66d889fb0" containerName="extract" Oct 30 00:09:24.684518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684388 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="712c67e6-6358-4c10-b540-f6f66d889fb0" containerName="extract" Oct 30 00:09:24.684518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684395 5001 cpu_manager.go:414] "containerMap: removing container" podUID="9c977822-ba67-47f8-9a59-fd265de65217" containerName="extract" Oct 30 00:09:24.684518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684399 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c977822-ba67-47f8-9a59-fd265de65217" containerName="extract" Oct 30 00:09:24.684518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684406 5001 cpu_manager.go:414] "containerMap: removing container" podUID="08d02afc-16dd-4010-9292-4d394739a984" containerName="pull" Oct 30 00:09:24.684518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684411 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08d02afc-16dd-4010-9292-4d394739a984" containerName="pull" Oct 30 00:09:24.684518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684417 5001 cpu_manager.go:414] "containerMap: removing container" podUID="08d02afc-16dd-4010-9292-4d394739a984" containerName="extract" Oct 30 00:09:24.684518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684422 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08d02afc-16dd-4010-9292-4d394739a984" containerName="extract" Oct 30 00:09:24.684518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684431 5001 cpu_manager.go:414] "containerMap: removing container" podUID="a78c8440-3039-46ab-b3c6-d0276d984f56" containerName="pull" Oct 30 00:09:24.684518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684435 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a78c8440-3039-46ab-b3c6-d0276d984f56" containerName="pull" Oct 30 00:09:24.684518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684441 5001 cpu_manager.go:414] "containerMap: removing container" podUID="7d1bbe05-289d-414a-8854-db469519e37e" containerName="pull" Oct 30 00:09:24.684518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684445 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d1bbe05-289d-414a-8854-db469519e37e" containerName="pull" Oct 30 00:09:24.684518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684451 5001 cpu_manager.go:414] "containerMap: removing container" podUID="556143f7-0f4b-456c-98c1-da699bc3c9c3" containerName="extract" Oct 30 00:09:24.684518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684455 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="556143f7-0f4b-456c-98c1-da699bc3c9c3" containerName="extract" Oct 30 00:09:24.684518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684462 5001 cpu_manager.go:414] "containerMap: removing container" podUID="5d34d892-f9ab-4bd2-920c-2c7305c7c57b" containerName="util" Oct 30 00:09:24.684518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684466 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d34d892-f9ab-4bd2-920c-2c7305c7c57b" containerName="util" Oct 30 00:09:24.684518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684473 5001 cpu_manager.go:414] "containerMap: removing container" podUID="bbffa049-445d-47f2-bcf6-6edd14cf25c1" containerName="extract" Oct 30 00:09:24.684518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684478 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbffa049-445d-47f2-bcf6-6edd14cf25c1" containerName="extract" Oct 30 00:09:24.684518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684483 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4484ecc5-88d4-4f8e-8244-a7b35f95bb3a" containerName="util" Oct 30 00:09:24.684518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684487 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4484ecc5-88d4-4f8e-8244-a7b35f95bb3a" containerName="util" Oct 30 00:09:24.684518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684493 5001 cpu_manager.go:414] "containerMap: removing container" podUID="7d1bbe05-289d-414a-8854-db469519e37e" containerName="extract" Oct 30 00:09:24.684518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684497 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d1bbe05-289d-414a-8854-db469519e37e" containerName="extract" Oct 30 00:09:24.684518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684505 5001 cpu_manager.go:414] "containerMap: removing container" podUID="95276631-3e7c-4d4b-a182-8b5d0fa319b8" containerName="util" Oct 30 00:09:24.684518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684509 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95276631-3e7c-4d4b-a182-8b5d0fa319b8" containerName="util" Oct 30 00:09:24.684518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684514 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4484ecc5-88d4-4f8e-8244-a7b35f95bb3a" containerName="extract" Oct 30 00:09:24.684518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684518 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4484ecc5-88d4-4f8e-8244-a7b35f95bb3a" containerName="extract" Oct 30 00:09:24.684518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684526 5001 cpu_manager.go:414] "containerMap: removing container" podUID="556143f7-0f4b-456c-98c1-da699bc3c9c3" containerName="util" Oct 30 00:09:24.684518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684531 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="556143f7-0f4b-456c-98c1-da699bc3c9c3" containerName="util" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684539 5001 cpu_manager.go:414] "containerMap: removing container" podUID="08d02afc-16dd-4010-9292-4d394739a984" containerName="util" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684544 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08d02afc-16dd-4010-9292-4d394739a984" containerName="util" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684549 5001 cpu_manager.go:414] "containerMap: removing container" podUID="556143f7-0f4b-456c-98c1-da699bc3c9c3" containerName="pull" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684554 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="556143f7-0f4b-456c-98c1-da699bc3c9c3" containerName="pull" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684561 5001 cpu_manager.go:414] "containerMap: removing container" podUID="9c977822-ba67-47f8-9a59-fd265de65217" containerName="pull" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684565 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c977822-ba67-47f8-9a59-fd265de65217" containerName="pull" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684574 5001 cpu_manager.go:414] "containerMap: removing container" podUID="bbffa049-445d-47f2-bcf6-6edd14cf25c1" containerName="util" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684578 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbffa049-445d-47f2-bcf6-6edd14cf25c1" containerName="util" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684584 5001 cpu_manager.go:414] "containerMap: removing container" podUID="5d34d892-f9ab-4bd2-920c-2c7305c7c57b" containerName="pull" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684588 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d34d892-f9ab-4bd2-920c-2c7305c7c57b" containerName="pull" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684594 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b44f71d9-0521-4d58-a1a4-00aa0b0b11f2" containerName="extract" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684599 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b44f71d9-0521-4d58-a1a4-00aa0b0b11f2" containerName="extract" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684606 5001 cpu_manager.go:414] "containerMap: removing container" podUID="9c977822-ba67-47f8-9a59-fd265de65217" containerName="util" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684610 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c977822-ba67-47f8-9a59-fd265de65217" containerName="util" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684615 5001 cpu_manager.go:414] "containerMap: removing container" podUID="bbffa049-445d-47f2-bcf6-6edd14cf25c1" containerName="pull" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684619 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbffa049-445d-47f2-bcf6-6edd14cf25c1" containerName="pull" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684623 5001 cpu_manager.go:414] "containerMap: removing container" podUID="95276631-3e7c-4d4b-a182-8b5d0fa319b8" containerName="extract" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684627 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95276631-3e7c-4d4b-a182-8b5d0fa319b8" containerName="extract" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684636 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4484ecc5-88d4-4f8e-8244-a7b35f95bb3a" containerName="pull" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684640 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4484ecc5-88d4-4f8e-8244-a7b35f95bb3a" containerName="pull" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684645 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b44f71d9-0521-4d58-a1a4-00aa0b0b11f2" containerName="util" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684649 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b44f71d9-0521-4d58-a1a4-00aa0b0b11f2" containerName="util" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684656 5001 cpu_manager.go:414] "containerMap: removing container" podUID="95276631-3e7c-4d4b-a182-8b5d0fa319b8" containerName="pull" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684660 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95276631-3e7c-4d4b-a182-8b5d0fa319b8" containerName="pull" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684665 5001 cpu_manager.go:414] "containerMap: removing container" podUID="712c67e6-6358-4c10-b540-f6f66d889fb0" containerName="pull" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684669 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="712c67e6-6358-4c10-b540-f6f66d889fb0" containerName="pull" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684675 5001 cpu_manager.go:414] "containerMap: removing container" podUID="5d34d892-f9ab-4bd2-920c-2c7305c7c57b" containerName="extract" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684680 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d34d892-f9ab-4bd2-920c-2c7305c7c57b" containerName="extract" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684686 5001 cpu_manager.go:414] "containerMap: removing container" podUID="a78c8440-3039-46ab-b3c6-d0276d984f56" containerName="extract" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684690 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a78c8440-3039-46ab-b3c6-d0276d984f56" containerName="extract" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684700 5001 cpu_manager.go:414] "containerMap: removing container" podUID="7d1bbe05-289d-414a-8854-db469519e37e" containerName="util" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684704 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d1bbe05-289d-414a-8854-db469519e37e" containerName="util" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684710 5001 cpu_manager.go:414] "containerMap: removing container" podUID="a78c8440-3039-46ab-b3c6-d0276d984f56" containerName="util" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684715 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a78c8440-3039-46ab-b3c6-d0276d984f56" containerName="util" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684760 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="556143f7-0f4b-456c-98c1-da699bc3c9c3" containerName="extract" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684767 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbffa049-445d-47f2-bcf6-6edd14cf25c1" containerName="extract" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684772 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d1bbe05-289d-414a-8854-db469519e37e" containerName="extract" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684778 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4484ecc5-88d4-4f8e-8244-a7b35f95bb3a" containerName="extract" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684784 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b44f71d9-0521-4d58-a1a4-00aa0b0b11f2" containerName="extract" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684790 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="95276631-3e7c-4d4b-a182-8b5d0fa319b8" containerName="extract" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684795 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d34d892-f9ab-4bd2-920c-2c7305c7c57b" containerName="extract" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684800 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a78c8440-3039-46ab-b3c6-d0276d984f56" containerName="extract" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684805 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c977822-ba67-47f8-9a59-fd265de65217" containerName="extract" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684810 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="712c67e6-6358-4c10-b540-f6f66d889fb0" containerName="extract" Oct 30 00:09:24.685428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.684814 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="08d02afc-16dd-4010-9292-4d394739a984" containerName="extract" Oct 30 00:09:24.687406 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.687390 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lcpns" Oct 30 00:09:24.707155 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.707137 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 00:09:24.708464 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.708437 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl6bf\" (UniqueName: \"kubernetes.io/projected/1132bdec-b38c-4db6-b832-272a81abffa8-kube-api-access-rl6bf\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lcpns\" (UID: \"1132bdec-b38c-4db6-b832-272a81abffa8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lcpns" Oct 30 00:09:24.708578 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.708497 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1132bdec-b38c-4db6-b832-272a81abffa8-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lcpns\" (UID: \"1132bdec-b38c-4db6-b832-272a81abffa8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lcpns" Oct 30 00:09:24.711625 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.711604 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-598zc\"" Oct 30 00:09:24.711625 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.711611 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:09:24.713817 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.713792 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lcpns"] Oct 30 00:09:24.809831 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.809801 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl6bf\" (UniqueName: \"kubernetes.io/projected/1132bdec-b38c-4db6-b832-272a81abffa8-kube-api-access-rl6bf\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lcpns\" (UID: \"1132bdec-b38c-4db6-b832-272a81abffa8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lcpns" Oct 30 00:09:24.810028 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.809853 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1132bdec-b38c-4db6-b832-272a81abffa8-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lcpns\" (UID: \"1132bdec-b38c-4db6-b832-272a81abffa8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lcpns" Oct 30 00:09:24.810228 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.810212 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1132bdec-b38c-4db6-b832-272a81abffa8-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lcpns\" (UID: \"1132bdec-b38c-4db6-b832-272a81abffa8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lcpns" Oct 30 00:09:24.830609 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.830581 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl6bf\" (UniqueName: \"kubernetes.io/projected/1132bdec-b38c-4db6-b832-272a81abffa8-kube-api-access-rl6bf\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lcpns\" (UID: \"1132bdec-b38c-4db6-b832-272a81abffa8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lcpns" Oct 30 00:09:24.997423 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:24.997345 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lcpns" Oct 30 00:09:25.323238 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:25.323203 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lcpns"] Oct 30 00:09:25.327140 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:09:25.327112 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1132bdec_b38c_4db6_b832_272a81abffa8.slice/crio-3b509a8ee8a32b05752f7f567d0a704c3ea24a7f7f908df033b23be03973fc5d WatchSource:0}: Error finding container 3b509a8ee8a32b05752f7f567d0a704c3ea24a7f7f908df033b23be03973fc5d: Status 404 returned error can't find the container with id 3b509a8ee8a32b05752f7f567d0a704c3ea24a7f7f908df033b23be03973fc5d Oct 30 00:09:25.688114 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:25.688038 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d4f7fd4c5-c5vm4" Oct 30 00:09:25.771360 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:25.771326 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lcpns" event={"ID":"1132bdec-b38c-4db6-b832-272a81abffa8","Type":"ContainerStarted","Data":"3b509a8ee8a32b05752f7f567d0a704c3ea24a7f7f908df033b23be03973fc5d"} Oct 30 00:09:27.783230 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:27.783194 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lcpns" event={"ID":"1132bdec-b38c-4db6-b832-272a81abffa8","Type":"ContainerStarted","Data":"0af8bcdd86fba6e2e6378702c04e56fff18071d50f0199a345df2c7f6fc72b69"} Oct 30 00:09:27.854814 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:27.854763 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lcpns" podStartSLOduration=2.362195878 podStartE2EDuration="3.854748606s" totalImagesPullingTime="1.492552728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:09:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:09:27.843193602 +0000 UTC m=+283.048694783" watchObservedRunningTime="2026-10-30 00:09:27.854748606 +0000 UTC m=+283.060249787" Oct 30 00:09:29.283132 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:29.283096 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-z5fc5"] Oct 30 00:09:29.286594 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:29.286575 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-z5fc5" Oct 30 00:09:29.312474 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:29.312450 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-z5fc5"] Oct 30 00:09:29.320321 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:29.320301 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-2jklr\"" Oct 30 00:09:29.349583 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:29.349549 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23b0993f-6678-4681-8c78-6c0a2feaabbd-tmp\") pod \"jobset-operator-674cd59dc5-z5fc5\" (UID: \"23b0993f-6678-4681-8c78-6c0a2feaabbd\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-z5fc5" Oct 30 00:09:29.349718 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:29.349588 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4fxn\" (UniqueName: \"kubernetes.io/projected/23b0993f-6678-4681-8c78-6c0a2feaabbd-kube-api-access-f4fxn\") pod \"jobset-operator-674cd59dc5-z5fc5\" (UID: \"23b0993f-6678-4681-8c78-6c0a2feaabbd\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-z5fc5" Oct 30 00:09:29.382718 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:29.382690 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 00:09:29.382848 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:29.382747 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:09:29.450265 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:29.450231 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23b0993f-6678-4681-8c78-6c0a2feaabbd-tmp\") pod \"jobset-operator-674cd59dc5-z5fc5\" (UID: \"23b0993f-6678-4681-8c78-6c0a2feaabbd\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-z5fc5" Oct 30 00:09:29.450428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:29.450272 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4fxn\" (UniqueName: \"kubernetes.io/projected/23b0993f-6678-4681-8c78-6c0a2feaabbd-kube-api-access-f4fxn\") pod \"jobset-operator-674cd59dc5-z5fc5\" (UID: \"23b0993f-6678-4681-8c78-6c0a2feaabbd\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-z5fc5" Oct 30 00:09:29.450624 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:29.450605 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23b0993f-6678-4681-8c78-6c0a2feaabbd-tmp\") pod \"jobset-operator-674cd59dc5-z5fc5\" (UID: \"23b0993f-6678-4681-8c78-6c0a2feaabbd\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-z5fc5" Oct 30 00:09:29.472210 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:29.472181 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4fxn\" (UniqueName: \"kubernetes.io/projected/23b0993f-6678-4681-8c78-6c0a2feaabbd-kube-api-access-f4fxn\") pod \"jobset-operator-674cd59dc5-z5fc5\" (UID: \"23b0993f-6678-4681-8c78-6c0a2feaabbd\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-z5fc5" Oct 30 00:09:29.611057 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:29.610961 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-z5fc5" Oct 30 00:09:29.909445 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:29.909363 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-z5fc5"] Oct 30 00:09:29.913227 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:09:29.913196 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23b0993f_6678_4681_8c78_6c0a2feaabbd.slice/crio-9676f3d5937d1e5b28a7acb44fee1e01e23a017ed74090ff05aa478b07ff8fce WatchSource:0}: Error finding container 9676f3d5937d1e5b28a7acb44fee1e01e23a017ed74090ff05aa478b07ff8fce: Status 404 returned error can't find the container with id 9676f3d5937d1e5b28a7acb44fee1e01e23a017ed74090ff05aa478b07ff8fce Oct 30 00:09:30.798609 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:30.798577 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-z5fc5" event={"ID":"23b0993f-6678-4681-8c78-6c0a2feaabbd","Type":"ContainerStarted","Data":"9676f3d5937d1e5b28a7acb44fee1e01e23a017ed74090ff05aa478b07ff8fce"} Oct 30 00:09:32.811573 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:32.811529 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-z5fc5" event={"ID":"23b0993f-6678-4681-8c78-6c0a2feaabbd","Type":"ContainerStarted","Data":"c458e3ebf4b0aec5a731fcd9d1f323b0c5c0b4ae1825e37786ed8d1887978772"} Oct 30 00:09:32.849626 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:32.849535 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-z5fc5" podStartSLOduration=1.461144432 podStartE2EDuration="3.849516835s" totalImagesPullingTime="2.388372403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:09:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:09:32.847394704 +0000 UTC m=+288.052895884" watchObservedRunningTime="2026-10-30 00:09:32.849516835 +0000 UTC m=+288.055018017" Oct 30 00:09:40.231954 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:40.231907 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-qhxpk"] Oct 30 00:09:40.253714 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:40.253687 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-qhxpk"] Oct 30 00:09:40.253875 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:40.253811 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-qhxpk" Oct 30 00:09:40.266193 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:40.266171 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-qd78b\"" Oct 30 00:09:40.266582 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:40.266569 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 00:09:40.276141 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:40.276120 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 00:09:40.343502 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:40.343464 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lnf2\" (UniqueName: \"kubernetes.io/projected/1e9c2ccc-5839-45fb-95d0-08619d6817a8-kube-api-access-5lnf2\") pod \"cert-manager-597b95cb84-qhxpk\" (UID: \"1e9c2ccc-5839-45fb-95d0-08619d6817a8\") " pod="cert-manager/cert-manager-597b95cb84-qhxpk" Oct 30 00:09:40.343704 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:40.343519 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e9c2ccc-5839-45fb-95d0-08619d6817a8-bound-sa-token\") pod \"cert-manager-597b95cb84-qhxpk\" (UID: \"1e9c2ccc-5839-45fb-95d0-08619d6817a8\") " pod="cert-manager/cert-manager-597b95cb84-qhxpk" Oct 30 00:09:40.444253 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:40.444217 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lnf2\" (UniqueName: \"kubernetes.io/projected/1e9c2ccc-5839-45fb-95d0-08619d6817a8-kube-api-access-5lnf2\") pod \"cert-manager-597b95cb84-qhxpk\" (UID: \"1e9c2ccc-5839-45fb-95d0-08619d6817a8\") " pod="cert-manager/cert-manager-597b95cb84-qhxpk" Oct 30 00:09:40.444464 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:40.444270 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e9c2ccc-5839-45fb-95d0-08619d6817a8-bound-sa-token\") pod \"cert-manager-597b95cb84-qhxpk\" (UID: \"1e9c2ccc-5839-45fb-95d0-08619d6817a8\") " pod="cert-manager/cert-manager-597b95cb84-qhxpk" Oct 30 00:09:40.465047 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:40.465019 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lnf2\" (UniqueName: \"kubernetes.io/projected/1e9c2ccc-5839-45fb-95d0-08619d6817a8-kube-api-access-5lnf2\") pod \"cert-manager-597b95cb84-qhxpk\" (UID: \"1e9c2ccc-5839-45fb-95d0-08619d6817a8\") " pod="cert-manager/cert-manager-597b95cb84-qhxpk" Oct 30 00:09:40.465197 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:40.465100 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e9c2ccc-5839-45fb-95d0-08619d6817a8-bound-sa-token\") pod \"cert-manager-597b95cb84-qhxpk\" (UID: \"1e9c2ccc-5839-45fb-95d0-08619d6817a8\") " pod="cert-manager/cert-manager-597b95cb84-qhxpk" Oct 30 00:09:40.576830 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:40.576783 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-qhxpk" Oct 30 00:09:40.874767 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:40.874689 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-qhxpk"] Oct 30 00:09:40.878690 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:09:40.878661 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e9c2ccc_5839_45fb_95d0_08619d6817a8.slice/crio-81fd7186a462cb443ca471af51b9a051ae0dba8f2c4b1796c533a6dcd05bc682 WatchSource:0}: Error finding container 81fd7186a462cb443ca471af51b9a051ae0dba8f2c4b1796c533a6dcd05bc682: Status 404 returned error can't find the container with id 81fd7186a462cb443ca471af51b9a051ae0dba8f2c4b1796c533a6dcd05bc682 Oct 30 00:09:41.861633 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:41.861596 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-qhxpk" event={"ID":"1e9c2ccc-5839-45fb-95d0-08619d6817a8","Type":"ContainerStarted","Data":"81fd7186a462cb443ca471af51b9a051ae0dba8f2c4b1796c533a6dcd05bc682"} Oct 30 00:09:42.741038 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:42.741004 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xm2mx"] Oct 30 00:09:42.744405 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:42.744384 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xm2mx" Oct 30 00:09:42.757690 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:42.757663 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xm2mx"] Oct 30 00:09:42.764839 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:42.764815 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 00:09:42.764980 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:42.764877 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wl68b\"" Oct 30 00:09:42.765068 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:42.765002 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 00:09:42.865165 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:42.865130 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29rw5\" (UniqueName: \"kubernetes.io/projected/8b2588a8-6823-4e6c-b637-818d0f327ad6-kube-api-access-29rw5\") pod \"kuadrant-operator-controller-manager-cf8469d7b-xm2mx\" (UID: \"8b2588a8-6823-4e6c-b637-818d0f327ad6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xm2mx" Oct 30 00:09:42.865636 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:42.865252 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b2588a8-6823-4e6c-b637-818d0f327ad6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-xm2mx\" (UID: \"8b2588a8-6823-4e6c-b637-818d0f327ad6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xm2mx" Oct 30 00:09:42.966745 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:42.966706 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b2588a8-6823-4e6c-b637-818d0f327ad6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-xm2mx\" (UID: \"8b2588a8-6823-4e6c-b637-818d0f327ad6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xm2mx" Oct 30 00:09:42.966948 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:42.966792 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29rw5\" (UniqueName: \"kubernetes.io/projected/8b2588a8-6823-4e6c-b637-818d0f327ad6-kube-api-access-29rw5\") pod \"kuadrant-operator-controller-manager-cf8469d7b-xm2mx\" (UID: \"8b2588a8-6823-4e6c-b637-818d0f327ad6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xm2mx" Oct 30 00:09:42.967215 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:42.967186 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b2588a8-6823-4e6c-b637-818d0f327ad6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-xm2mx\" (UID: \"8b2588a8-6823-4e6c-b637-818d0f327ad6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xm2mx" Oct 30 00:09:42.988108 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:42.988069 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29rw5\" (UniqueName: \"kubernetes.io/projected/8b2588a8-6823-4e6c-b637-818d0f327ad6-kube-api-access-29rw5\") pod \"kuadrant-operator-controller-manager-cf8469d7b-xm2mx\" (UID: \"8b2588a8-6823-4e6c-b637-818d0f327ad6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xm2mx" Oct 30 00:09:43.060510 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:43.060475 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xm2mx" Oct 30 00:09:43.407941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:43.407860 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xm2mx"] Oct 30 00:09:43.411470 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:09:43.411447 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b2588a8_6823_4e6c_b637_818d0f327ad6.slice/crio-525639b49a07963b4ac4f9a2ced7feba9337be1942dee2d80d00a896198ab6fd WatchSource:0}: Error finding container 525639b49a07963b4ac4f9a2ced7feba9337be1942dee2d80d00a896198ab6fd: Status 404 returned error can't find the container with id 525639b49a07963b4ac4f9a2ced7feba9337be1942dee2d80d00a896198ab6fd Oct 30 00:09:43.875335 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:43.875301 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xm2mx" event={"ID":"8b2588a8-6823-4e6c-b637-818d0f327ad6","Type":"ContainerStarted","Data":"525639b49a07963b4ac4f9a2ced7feba9337be1942dee2d80d00a896198ab6fd"} Oct 30 00:09:43.876821 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:43.876796 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-qhxpk" event={"ID":"1e9c2ccc-5839-45fb-95d0-08619d6817a8","Type":"ContainerStarted","Data":"181dff6c7e556713de1014bcbfbf1d938e326a055cab720928a2b9e19935e112"} Oct 30 00:09:43.913101 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:43.913037 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-qhxpk" podStartSLOduration=1.455155434 podStartE2EDuration="3.913018077s" totalImagesPullingTime="2.457862643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:09:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:09:43.911241685 +0000 UTC m=+299.116742866" watchObservedRunningTime="2026-10-30 00:09:43.913018077 +0000 UTC m=+299.118519264" Oct 30 00:09:44.965701 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:44.965670 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8gb5_9fa688d8-29ca-4c1f-926a-f1a4099e6b44/console-operator/2.log" Oct 30 00:09:44.968117 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:44.968094 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8gb5_9fa688d8-29ca-4c1f-926a-f1a4099e6b44/console-operator/2.log" Oct 30 00:09:45.082476 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:45.082440 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-vxqwt"] Oct 30 00:09:45.086587 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:45.086562 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vxqwt" Oct 30 00:09:45.103040 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:45.103010 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 00:09:45.113069 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:45.113038 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-qh7dq\"" Oct 30 00:09:45.113616 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:45.113595 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-vxqwt"] Oct 30 00:09:45.186389 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:45.186331 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4lpc\" (UniqueName: \"kubernetes.io/projected/0aa4310d-ba64-4c3a-b1ca-0d5a8d1522fb-kube-api-access-l4lpc\") pod \"dns-operator-controller-manager-8557cbdb84-vxqwt\" (UID: \"0aa4310d-ba64-4c3a-b1ca-0d5a8d1522fb\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vxqwt" Oct 30 00:09:45.287820 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:45.287781 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4lpc\" (UniqueName: \"kubernetes.io/projected/0aa4310d-ba64-4c3a-b1ca-0d5a8d1522fb-kube-api-access-l4lpc\") pod \"dns-operator-controller-manager-8557cbdb84-vxqwt\" (UID: \"0aa4310d-ba64-4c3a-b1ca-0d5a8d1522fb\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vxqwt" Oct 30 00:09:45.317798 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:45.317769 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4lpc\" (UniqueName: \"kubernetes.io/projected/0aa4310d-ba64-4c3a-b1ca-0d5a8d1522fb-kube-api-access-l4lpc\") pod \"dns-operator-controller-manager-8557cbdb84-vxqwt\" (UID: \"0aa4310d-ba64-4c3a-b1ca-0d5a8d1522fb\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vxqwt" Oct 30 00:09:45.406283 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:45.406234 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vxqwt" Oct 30 00:09:45.719827 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:45.719751 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-vxqwt"] Oct 30 00:09:45.724296 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:09:45.724262 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aa4310d_ba64_4c3a_b1ca_0d5a8d1522fb.slice/crio-ca8fc5e7a412323ce8bd137783890c4c1d605b537d5d1c92bbe82fbba8471b09 WatchSource:0}: Error finding container ca8fc5e7a412323ce8bd137783890c4c1d605b537d5d1c92bbe82fbba8471b09: Status 404 returned error can't find the container with id ca8fc5e7a412323ce8bd137783890c4c1d605b537d5d1c92bbe82fbba8471b09 Oct 30 00:09:45.895218 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:45.895171 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vxqwt" event={"ID":"0aa4310d-ba64-4c3a-b1ca-0d5a8d1522fb","Type":"ContainerStarted","Data":"ca8fc5e7a412323ce8bd137783890c4c1d605b537d5d1c92bbe82fbba8471b09"} Oct 30 00:09:47.326860 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:09:47.326822 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:09:47.479889 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:47.479864 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-xr4lb"] Oct 30 00:09:47.485950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:47.484842 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xr4lb" Oct 30 00:09:47.500315 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:47.500292 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-n5x59\"" Oct 30 00:09:47.503788 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:47.503720 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-xr4lb"] Oct 30 00:09:47.609992 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:47.609891 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqcd7\" (UniqueName: \"kubernetes.io/projected/75b06ad5-f34b-4a66-b00a-50325ca9fc0b-kube-api-access-zqcd7\") pod \"limitador-operator-controller-manager-5778bb4449-xr4lb\" (UID: \"75b06ad5-f34b-4a66-b00a-50325ca9fc0b\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xr4lb" Oct 30 00:09:47.711320 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:47.710930 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqcd7\" (UniqueName: \"kubernetes.io/projected/75b06ad5-f34b-4a66-b00a-50325ca9fc0b-kube-api-access-zqcd7\") pod \"limitador-operator-controller-manager-5778bb4449-xr4lb\" (UID: \"75b06ad5-f34b-4a66-b00a-50325ca9fc0b\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xr4lb" Oct 30 00:09:47.735073 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:47.735041 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqcd7\" (UniqueName: \"kubernetes.io/projected/75b06ad5-f34b-4a66-b00a-50325ca9fc0b-kube-api-access-zqcd7\") pod \"limitador-operator-controller-manager-5778bb4449-xr4lb\" (UID: \"75b06ad5-f34b-4a66-b00a-50325ca9fc0b\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xr4lb" Oct 30 00:09:47.823177 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:47.823138 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xr4lb" Oct 30 00:09:47.909098 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:47.909067 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vxqwt" event={"ID":"0aa4310d-ba64-4c3a-b1ca-0d5a8d1522fb","Type":"ContainerStarted","Data":"ebafa8fa23347f76e2c83f7ea1ebd3b22c47b589e81e37c96cc23304e9dbf06e"} Oct 30 00:09:47.909284 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:47.909264 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vxqwt" Oct 30 00:09:47.947894 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:47.947841 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vxqwt" podStartSLOduration=1.201265443 podStartE2EDuration="2.947825853s" totalImagesPullingTime="1.74656041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:09:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:09:47.947160621 +0000 UTC m=+303.152661801" watchObservedRunningTime="2026-10-30 00:09:47.947825853 +0000 UTC m=+303.153327035" Oct 30 00:09:48.136426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:48.136349 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-xr4lb"] Oct 30 00:09:48.139962 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:09:48.139936 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b06ad5_f34b_4a66_b00a_50325ca9fc0b.slice/crio-35d35243e005648cdbc685604921614684c56dcc17711b913269428f51d35623 WatchSource:0}: Error finding container 35d35243e005648cdbc685604921614684c56dcc17711b913269428f51d35623: Status 404 returned error can't find the container with id 35d35243e005648cdbc685604921614684c56dcc17711b913269428f51d35623 Oct 30 00:09:48.922006 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:48.921969 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xr4lb" event={"ID":"75b06ad5-f34b-4a66-b00a-50325ca9fc0b","Type":"ContainerStarted","Data":"35d35243e005648cdbc685604921614684c56dcc17711b913269428f51d35623"} Oct 30 00:09:48.923411 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:48.923378 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xm2mx" event={"ID":"8b2588a8-6823-4e6c-b637-818d0f327ad6","Type":"ContainerStarted","Data":"7e0b5350932fdc5c4695239d40e453b8001493ce57d8b790e9f0364df9e56c7b"} Oct 30 00:09:48.923542 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:48.923434 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xm2mx" Oct 30 00:09:49.000255 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:49.000199 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xm2mx" podStartSLOduration=2.453175534 podStartE2EDuration="7.000182544s" totalImagesPullingTime="4.54700701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:09:48.987351168 +0000 UTC m=+304.192852348" watchObservedRunningTime="2026-10-30 00:09:49.000182544 +0000 UTC m=+304.205683726" Oct 30 00:09:49.930406 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:49.930365 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xr4lb" event={"ID":"75b06ad5-f34b-4a66-b00a-50325ca9fc0b","Type":"ContainerStarted","Data":"1b87829368cb16796712251aa0a32af5360c1ddcce65201b1a4cf90f110661d7"} Oct 30 00:09:49.930894 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:49.930588 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xr4lb" Oct 30 00:09:49.975795 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:49.975695 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xr4lb" podStartSLOduration=1.394002279 podStartE2EDuration="2.975680253s" totalImagesPullingTime="1.581677974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:09:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:09:49.973467733 +0000 UTC m=+305.178968915" watchObservedRunningTime="2026-10-30 00:09:49.975680253 +0000 UTC m=+305.181181433" Oct 30 00:09:51.886113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:51.886078 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd"] Oct 30 00:09:51.889833 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:51.889811 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" Oct 30 00:09:51.921135 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:51.921106 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd"] Oct 30 00:09:51.933714 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:51.933681 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-p2xxb\"" Oct 30 00:09:51.947630 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:51.947599 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmrfs\" (UniqueName: \"kubernetes.io/projected/8f5c5c45-659f-498f-8585-672fade7b41d-kube-api-access-vmrfs\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd\" (UID: \"8f5c5c45-659f-498f-8585-672fade7b41d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" Oct 30 00:09:51.947768 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:51.947646 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5c5c45-659f-498f-8585-672fade7b41d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd\" (UID: \"8f5c5c45-659f-498f-8585-672fade7b41d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" Oct 30 00:09:51.947768 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:51.947712 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f5c5c45-659f-498f-8585-672fade7b41d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd\" (UID: \"8f5c5c45-659f-498f-8585-672fade7b41d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" Oct 30 00:09:51.947768 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:51.947751 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f5c5c45-659f-498f-8585-672fade7b41d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd\" (UID: \"8f5c5c45-659f-498f-8585-672fade7b41d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" Oct 30 00:09:51.947965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:51.947852 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f5c5c45-659f-498f-8585-672fade7b41d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd\" (UID: \"8f5c5c45-659f-498f-8585-672fade7b41d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" Oct 30 00:09:51.947965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:51.947904 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5c5c45-659f-498f-8585-672fade7b41d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd\" (UID: \"8f5c5c45-659f-498f-8585-672fade7b41d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" Oct 30 00:09:51.947965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:51.947944 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f5c5c45-659f-498f-8585-672fade7b41d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd\" (UID: \"8f5c5c45-659f-498f-8585-672fade7b41d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" Oct 30 00:09:51.948139 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:51.947965 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f5c5c45-659f-498f-8585-672fade7b41d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd\" (UID: \"8f5c5c45-659f-498f-8585-672fade7b41d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" Oct 30 00:09:51.948139 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:51.948057 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f5c5c45-659f-498f-8585-672fade7b41d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd\" (UID: \"8f5c5c45-659f-498f-8585-672fade7b41d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" Oct 30 00:09:52.049489 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:52.049452 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5c5c45-659f-498f-8585-672fade7b41d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd\" (UID: \"8f5c5c45-659f-498f-8585-672fade7b41d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" Oct 30 00:09:52.049672 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:52.049499 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f5c5c45-659f-498f-8585-672fade7b41d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd\" (UID: \"8f5c5c45-659f-498f-8585-672fade7b41d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" Oct 30 00:09:52.049672 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:52.049515 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f5c5c45-659f-498f-8585-672fade7b41d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd\" (UID: \"8f5c5c45-659f-498f-8585-672fade7b41d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" Oct 30 00:09:52.049672 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:52.049544 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f5c5c45-659f-498f-8585-672fade7b41d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd\" (UID: \"8f5c5c45-659f-498f-8585-672fade7b41d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" Oct 30 00:09:52.049672 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:52.049588 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmrfs\" (UniqueName: \"kubernetes.io/projected/8f5c5c45-659f-498f-8585-672fade7b41d-kube-api-access-vmrfs\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd\" (UID: \"8f5c5c45-659f-498f-8585-672fade7b41d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" Oct 30 00:09:52.049672 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:52.049640 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5c5c45-659f-498f-8585-672fade7b41d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd\" (UID: \"8f5c5c45-659f-498f-8585-672fade7b41d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" Oct 30 00:09:52.049672 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:52.049673 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f5c5c45-659f-498f-8585-672fade7b41d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd\" (UID: \"8f5c5c45-659f-498f-8585-672fade7b41d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" Oct 30 00:09:52.050009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:52.049956 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5c5c45-659f-498f-8585-672fade7b41d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd\" (UID: \"8f5c5c45-659f-498f-8585-672fade7b41d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" Oct 30 00:09:52.050009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:52.049984 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f5c5c45-659f-498f-8585-672fade7b41d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd\" (UID: \"8f5c5c45-659f-498f-8585-672fade7b41d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" Oct 30 00:09:52.050104 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:52.050011 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f5c5c45-659f-498f-8585-672fade7b41d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd\" (UID: \"8f5c5c45-659f-498f-8585-672fade7b41d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" Oct 30 00:09:52.050167 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:52.050153 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f5c5c45-659f-498f-8585-672fade7b41d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd\" (UID: \"8f5c5c45-659f-498f-8585-672fade7b41d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" Oct 30 00:09:52.050220 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:52.050157 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f5c5c45-659f-498f-8585-672fade7b41d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd\" (UID: \"8f5c5c45-659f-498f-8585-672fade7b41d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" Oct 30 00:09:52.050271 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:52.050217 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5c5c45-659f-498f-8585-672fade7b41d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd\" (UID: \"8f5c5c45-659f-498f-8585-672fade7b41d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" Oct 30 00:09:52.050600 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:52.050580 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f5c5c45-659f-498f-8585-672fade7b41d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd\" (UID: \"8f5c5c45-659f-498f-8585-672fade7b41d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" Oct 30 00:09:52.052012 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:52.051991 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f5c5c45-659f-498f-8585-672fade7b41d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd\" (UID: \"8f5c5c45-659f-498f-8585-672fade7b41d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" Oct 30 00:09:52.052258 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:52.052236 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f5c5c45-659f-498f-8585-672fade7b41d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd\" (UID: \"8f5c5c45-659f-498f-8585-672fade7b41d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" Oct 30 00:09:52.070845 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:52.070817 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f5c5c45-659f-498f-8585-672fade7b41d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd\" (UID: \"8f5c5c45-659f-498f-8585-672fade7b41d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" Oct 30 00:09:52.072930 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:52.072897 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmrfs\" (UniqueName: \"kubernetes.io/projected/8f5c5c45-659f-498f-8585-672fade7b41d-kube-api-access-vmrfs\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd\" (UID: \"8f5c5c45-659f-498f-8585-672fade7b41d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" Oct 30 00:09:52.204422 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:52.204341 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" Oct 30 00:09:52.522232 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:52.522198 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd"] Oct 30 00:09:52.525668 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:09:52.525640 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f5c5c45_659f_498f_8585_672fade7b41d.slice/crio-70af57d373a0946f6e1032e875c8f759dca3dcae7200e8ab677ba597a8d1cea5 WatchSource:0}: Error finding container 70af57d373a0946f6e1032e875c8f759dca3dcae7200e8ab677ba597a8d1cea5: Status 404 returned error can't find the container with id 70af57d373a0946f6e1032e875c8f759dca3dcae7200e8ab677ba597a8d1cea5 Oct 30 00:09:52.527881 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:52.527847 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:09:52.527969 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:52.527926 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:09:52.528025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:52.527972 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:09:52.947207 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:52.947120 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" event={"ID":"8f5c5c45-659f-498f-8585-672fade7b41d","Type":"ContainerStarted","Data":"e0983d335f276346efb5ca95ee29d17bbf24df2cf7cb05ea2dfa906b860fc2e0"} Oct 30 00:09:52.947207 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:52.947177 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" event={"ID":"8f5c5c45-659f-498f-8585-672fade7b41d","Type":"ContainerStarted","Data":"70af57d373a0946f6e1032e875c8f759dca3dcae7200e8ab677ba597a8d1cea5"} Oct 30 00:09:52.988654 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:52.988601 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" podStartSLOduration=1.9885865090000001 podStartE2EDuration="1.988586509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:09:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:09:52.986852972 +0000 UTC m=+308.192354153" watchObservedRunningTime="2026-10-30 00:09:52.988586509 +0000 UTC m=+308.194087733" Oct 30 00:09:53.204727 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:53.204635 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" Oct 30 00:09:53.209620 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:53.209596 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" Oct 30 00:09:53.952893 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:53.952858 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" Oct 30 00:09:53.953810 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:53.953789 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd" Oct 30 00:09:55.892346 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:55.892271 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-4n6qq"] Oct 30 00:09:55.895905 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:55.895882 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4n6qq" Oct 30 00:09:55.915824 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:55.915803 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-g28l2\"" Oct 30 00:09:55.924951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:55.924926 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 00:09:55.925158 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:55.925070 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 00:09:55.945520 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:55.945498 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 00:09:55.947612 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:55.947586 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-4n6qq"] Oct 30 00:09:55.985567 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:55.985539 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3febb7a0-47fa-4545-b76c-977789498353-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-4n6qq\" (UID: \"3febb7a0-47fa-4545-b76c-977789498353\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4n6qq" Oct 30 00:09:55.985698 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:55.985582 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3febb7a0-47fa-4545-b76c-977789498353-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-4n6qq\" (UID: \"3febb7a0-47fa-4545-b76c-977789498353\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4n6qq" Oct 30 00:09:55.985698 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:55.985676 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln7dr\" (UniqueName: \"kubernetes.io/projected/3febb7a0-47fa-4545-b76c-977789498353-kube-api-access-ln7dr\") pod \"kserve-module-controller-manager-7566cf5fc8-4n6qq\" (UID: \"3febb7a0-47fa-4545-b76c-977789498353\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4n6qq" Oct 30 00:09:55.985800 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:55.985735 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3febb7a0-47fa-4545-b76c-977789498353-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-4n6qq\" (UID: \"3febb7a0-47fa-4545-b76c-977789498353\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4n6qq" Oct 30 00:09:56.086251 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:56.086219 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3febb7a0-47fa-4545-b76c-977789498353-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-4n6qq\" (UID: \"3febb7a0-47fa-4545-b76c-977789498353\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4n6qq" Oct 30 00:09:56.086428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:56.086273 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3febb7a0-47fa-4545-b76c-977789498353-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-4n6qq\" (UID: \"3febb7a0-47fa-4545-b76c-977789498353\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4n6qq" Oct 30 00:09:56.086428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:56.086333 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln7dr\" (UniqueName: \"kubernetes.io/projected/3febb7a0-47fa-4545-b76c-977789498353-kube-api-access-ln7dr\") pod \"kserve-module-controller-manager-7566cf5fc8-4n6qq\" (UID: \"3febb7a0-47fa-4545-b76c-977789498353\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4n6qq" Oct 30 00:09:56.086428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:56.086374 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3febb7a0-47fa-4545-b76c-977789498353-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-4n6qq\" (UID: \"3febb7a0-47fa-4545-b76c-977789498353\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4n6qq" Oct 30 00:09:56.086800 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:56.086769 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3febb7a0-47fa-4545-b76c-977789498353-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-4n6qq\" (UID: \"3febb7a0-47fa-4545-b76c-977789498353\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4n6qq" Oct 30 00:09:56.086949 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:56.086845 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3febb7a0-47fa-4545-b76c-977789498353-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-4n6qq\" (UID: \"3febb7a0-47fa-4545-b76c-977789498353\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4n6qq" Oct 30 00:09:56.087162 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:56.087144 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3febb7a0-47fa-4545-b76c-977789498353-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-4n6qq\" (UID: \"3febb7a0-47fa-4545-b76c-977789498353\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4n6qq" Oct 30 00:09:56.106982 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:56.106961 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln7dr\" (UniqueName: \"kubernetes.io/projected/3febb7a0-47fa-4545-b76c-977789498353-kube-api-access-ln7dr\") pod \"kserve-module-controller-manager-7566cf5fc8-4n6qq\" (UID: \"3febb7a0-47fa-4545-b76c-977789498353\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4n6qq" Oct 30 00:09:56.209977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:56.209861 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4n6qq" Oct 30 00:09:56.517347 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:56.517317 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-4n6qq"] Oct 30 00:09:56.520857 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:09:56.520829 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3febb7a0_47fa_4545_b76c_977789498353.slice/crio-75a1a3c89596bd77c3e84a64d05c7cd8a4259177a99c24f444957396a259be3f WatchSource:0}: Error finding container 75a1a3c89596bd77c3e84a64d05c7cd8a4259177a99c24f444957396a259be3f: Status 404 returned error can't find the container with id 75a1a3c89596bd77c3e84a64d05c7cd8a4259177a99c24f444957396a259be3f Oct 30 00:09:56.523367 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:56.523347 5001 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:09:56.969885 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:56.969803 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4n6qq" event={"ID":"3febb7a0-47fa-4545-b76c-977789498353","Type":"ContainerStarted","Data":"75a1a3c89596bd77c3e84a64d05c7cd8a4259177a99c24f444957396a259be3f"} Oct 30 00:09:58.925691 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:58.925659 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vxqwt" Oct 30 00:09:58.982305 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:58.982215 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4n6qq" event={"ID":"3febb7a0-47fa-4545-b76c-977789498353","Type":"ContainerStarted","Data":"77fb49a7245c9250222fd1f7a88c1f0a80dc41aa4ce8e9b9ab9273f55a12bc84"} Oct 30 00:09:58.982305 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:58.982299 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4n6qq" Oct 30 00:09:59.021290 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:59.021240 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4n6qq" podStartSLOduration=1.843444187 podStartE2EDuration="4.02122621s" totalImagesPullingTime="2.177782023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:09:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:09:59.019439812 +0000 UTC m=+314.224940994" watchObservedRunningTime="2026-10-30 00:09:59.02122621 +0000 UTC m=+314.226727390" Oct 30 00:09:59.933004 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:09:59.932976 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xm2mx" Oct 30 00:10:00.937097 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:00.937069 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xr4lb" Oct 30 00:10:17.321966 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:10:17.321925 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:10:17.849698 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:17.849662 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xm2mx"] Oct 30 00:10:17.850022 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:17.849971 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xm2mx" podUID="8b2588a8-6823-4e6c-b637-818d0f327ad6" containerName="manager" containerID="cri-o://7e0b5350932fdc5c4695239d40e453b8001493ce57d8b790e9f0364df9e56c7b" gracePeriod=10 Oct 30 00:10:18.089561 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:18.089530 5001 generic.go:356] "Generic (PLEG): container finished" podID="8b2588a8-6823-4e6c-b637-818d0f327ad6" containerID="7e0b5350932fdc5c4695239d40e453b8001493ce57d8b790e9f0364df9e56c7b" exitCode=0 Oct 30 00:10:18.089728 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:18.089604 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xm2mx" event={"ID":"8b2588a8-6823-4e6c-b637-818d0f327ad6","Type":"ContainerDied","Data":"7e0b5350932fdc5c4695239d40e453b8001493ce57d8b790e9f0364df9e56c7b"} Oct 30 00:10:18.134535 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:18.134508 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xm2mx" Oct 30 00:10:18.180872 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:18.180833 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b2588a8-6823-4e6c-b637-818d0f327ad6-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b2588a8-6823-4e6c-b637-818d0f327ad6-extensions-socket-volume\") pod \"8b2588a8-6823-4e6c-b637-818d0f327ad6\" (UID: \"8b2588a8-6823-4e6c-b637-818d0f327ad6\") " Oct 30 00:10:18.181620 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:18.181588 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b2588a8-6823-4e6c-b637-818d0f327ad6-extensions-socket-volume" pod "8b2588a8-6823-4e6c-b637-818d0f327ad6" (UID: "8b2588a8-6823-4e6c-b637-818d0f327ad6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:10:18.182162 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:18.181877 5001 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b2588a8-6823-4e6c-b637-818d0f327ad6-extensions-socket-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:10:18.283112 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:18.283075 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b2588a8-6823-4e6c-b637-818d0f327ad6-kube-api-access-29rw5\" (UniqueName: \"kubernetes.io/projected/8b2588a8-6823-4e6c-b637-818d0f327ad6-kube-api-access-29rw5\") pod \"8b2588a8-6823-4e6c-b637-818d0f327ad6\" (UID: \"8b2588a8-6823-4e6c-b637-818d0f327ad6\") " Oct 30 00:10:18.285265 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:18.285237 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b2588a8-6823-4e6c-b637-818d0f327ad6-kube-api-access-29rw5" pod "8b2588a8-6823-4e6c-b637-818d0f327ad6" (UID: "8b2588a8-6823-4e6c-b637-818d0f327ad6"). InnerVolumeSpecName "kube-api-access-29rw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:10:18.384357 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:18.384273 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29rw5\" (UniqueName: \"kubernetes.io/projected/8b2588a8-6823-4e6c-b637-818d0f327ad6-kube-api-access-29rw5\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:10:19.096053 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:19.096021 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xm2mx" Oct 30 00:10:19.096241 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:19.096059 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xm2mx" event={"ID":"8b2588a8-6823-4e6c-b637-818d0f327ad6","Type":"ContainerDied","Data":"525639b49a07963b4ac4f9a2ced7feba9337be1942dee2d80d00a896198ab6fd"} Oct 30 00:10:19.096241 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:19.096127 5001 scope.go:122] "RemoveContainer" containerID="7e0b5350932fdc5c4695239d40e453b8001493ce57d8b790e9f0364df9e56c7b" Oct 30 00:10:19.162377 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:19.162339 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xm2mx"] Oct 30 00:10:19.172389 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:19.172360 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xm2mx"] Oct 30 00:10:19.989473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:19.989438 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4n6qq" Oct 30 00:10:20.915942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:20.915880 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b2588a8-6823-4e6c-b637-818d0f327ad6" path="/var/lib/kubelet/pods/8b2588a8-6823-4e6c-b637-818d0f327ad6/volumes" Oct 30 00:10:37.584223 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:37.584188 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4shsq"] Oct 30 00:10:37.584760 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:37.584734 5001 cpu_manager.go:414] "containerMap: removing container" podUID="8b2588a8-6823-4e6c-b637-818d0f327ad6" containerName="manager" Oct 30 00:10:37.584760 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:37.584760 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b2588a8-6823-4e6c-b637-818d0f327ad6" containerName="manager" Oct 30 00:10:37.585004 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:37.584849 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b2588a8-6823-4e6c-b637-818d0f327ad6" containerName="manager" Oct 30 00:10:37.589702 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:37.589681 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-4shsq" Oct 30 00:10:37.602986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:37.602900 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 00:10:37.612858 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:37.612836 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hsh2c\"" Oct 30 00:10:37.615478 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:37.615454 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4shsq"] Oct 30 00:10:37.642329 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:37.642304 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-z72x7"] Oct 30 00:10:37.644847 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:37.644824 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-z72x7" Oct 30 00:10:37.658020 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:37.657997 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b0ae782c-47d7-4895-9378-bbab2fc5ecfa-config-file\") pod \"limitador-limitador-74dfcfb894-4shsq\" (UID: \"b0ae782c-47d7-4895-9378-bbab2fc5ecfa\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-4shsq" Oct 30 00:10:37.658137 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:37.658039 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz68d\" (UniqueName: \"kubernetes.io/projected/b0ae782c-47d7-4895-9378-bbab2fc5ecfa-kube-api-access-kz68d\") pod \"limitador-limitador-74dfcfb894-4shsq\" (UID: \"b0ae782c-47d7-4895-9378-bbab2fc5ecfa\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-4shsq" Oct 30 00:10:37.665248 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:37.665219 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-z72x7"] Oct 30 00:10:37.703134 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:37.703065 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 00:10:37.703322 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:37.703220 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 00:10:37.725833 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:37.725804 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4shsq"] Oct 30 00:10:37.758464 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:37.758420 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/449bf72c-2ac6-42f7-8174-1e51e4c792cc-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-z72x7\" (UID: \"449bf72c-2ac6-42f7-8174-1e51e4c792cc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-z72x7" Oct 30 00:10:37.758637 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:37.758486 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b0ae782c-47d7-4895-9378-bbab2fc5ecfa-config-file\") pod \"limitador-limitador-74dfcfb894-4shsq\" (UID: \"b0ae782c-47d7-4895-9378-bbab2fc5ecfa\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-4shsq" Oct 30 00:10:37.758637 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:37.758584 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz68d\" (UniqueName: \"kubernetes.io/projected/b0ae782c-47d7-4895-9378-bbab2fc5ecfa-kube-api-access-kz68d\") pod \"limitador-limitador-74dfcfb894-4shsq\" (UID: \"b0ae782c-47d7-4895-9378-bbab2fc5ecfa\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-4shsq" Oct 30 00:10:37.758762 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:37.758631 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/449bf72c-2ac6-42f7-8174-1e51e4c792cc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-z72x7\" (UID: \"449bf72c-2ac6-42f7-8174-1e51e4c792cc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-z72x7" Oct 30 00:10:37.758762 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:37.758657 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7c2v\" (UniqueName: \"kubernetes.io/projected/449bf72c-2ac6-42f7-8174-1e51e4c792cc-kube-api-access-d7c2v\") pod \"kuadrant-console-plugin-66c4fb575d-z72x7\" (UID: \"449bf72c-2ac6-42f7-8174-1e51e4c792cc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-z72x7" Oct 30 00:10:37.759222 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:37.759199 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b0ae782c-47d7-4895-9378-bbab2fc5ecfa-config-file\") pod \"limitador-limitador-74dfcfb894-4shsq\" (UID: \"b0ae782c-47d7-4895-9378-bbab2fc5ecfa\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-4shsq" Oct 30 00:10:37.779332 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:37.779303 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz68d\" (UniqueName: \"kubernetes.io/projected/b0ae782c-47d7-4895-9378-bbab2fc5ecfa-kube-api-access-kz68d\") pod \"limitador-limitador-74dfcfb894-4shsq\" (UID: \"b0ae782c-47d7-4895-9378-bbab2fc5ecfa\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-4shsq" Oct 30 00:10:37.859701 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:37.859617 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/449bf72c-2ac6-42f7-8174-1e51e4c792cc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-z72x7\" (UID: \"449bf72c-2ac6-42f7-8174-1e51e4c792cc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-z72x7" Oct 30 00:10:37.859701 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:37.859658 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7c2v\" (UniqueName: \"kubernetes.io/projected/449bf72c-2ac6-42f7-8174-1e51e4c792cc-kube-api-access-d7c2v\") pod \"kuadrant-console-plugin-66c4fb575d-z72x7\" (UID: \"449bf72c-2ac6-42f7-8174-1e51e4c792cc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-z72x7" Oct 30 00:10:37.859701 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:37.859696 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/449bf72c-2ac6-42f7-8174-1e51e4c792cc-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-z72x7\" (UID: \"449bf72c-2ac6-42f7-8174-1e51e4c792cc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-z72x7" Oct 30 00:10:37.860383 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:37.860364 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/449bf72c-2ac6-42f7-8174-1e51e4c792cc-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-z72x7\" (UID: \"449bf72c-2ac6-42f7-8174-1e51e4c792cc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-z72x7" Oct 30 00:10:37.862142 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:37.862124 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/449bf72c-2ac6-42f7-8174-1e51e4c792cc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-z72x7\" (UID: \"449bf72c-2ac6-42f7-8174-1e51e4c792cc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-z72x7" Oct 30 00:10:37.880214 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:37.880189 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7c2v\" (UniqueName: \"kubernetes.io/projected/449bf72c-2ac6-42f7-8174-1e51e4c792cc-kube-api-access-d7c2v\") pod \"kuadrant-console-plugin-66c4fb575d-z72x7\" (UID: \"449bf72c-2ac6-42f7-8174-1e51e4c792cc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-z72x7" Oct 30 00:10:37.902824 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:37.902805 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-4shsq" Oct 30 00:10:37.958759 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:37.958732 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-z72x7" Oct 30 00:10:38.200340 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:38.200261 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4shsq"] Oct 30 00:10:38.204183 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:10:38.204155 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ae782c_47d7_4895_9378_bbab2fc5ecfa.slice/crio-bd4dafede53d813a64e4af12f1f507be8643efb305fc6a46a9dda065fe8c706a WatchSource:0}: Error finding container bd4dafede53d813a64e4af12f1f507be8643efb305fc6a46a9dda065fe8c706a: Status 404 returned error can't find the container with id bd4dafede53d813a64e4af12f1f507be8643efb305fc6a46a9dda065fe8c706a Oct 30 00:10:38.261351 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:38.261318 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-z72x7"] Oct 30 00:10:38.264987 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:10:38.264950 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod449bf72c_2ac6_42f7_8174_1e51e4c792cc.slice/crio-c568d47d118db3754bf2b2bd10930833cf031159ea8202e3d46e5461f563d716 WatchSource:0}: Error finding container c568d47d118db3754bf2b2bd10930833cf031159ea8202e3d46e5461f563d716: Status 404 returned error can't find the container with id c568d47d118db3754bf2b2bd10930833cf031159ea8202e3d46e5461f563d716 Oct 30 00:10:38.428024 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:38.427988 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-c5jln"] Oct 30 00:10:38.430939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:38.430896 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-c5jln" Oct 30 00:10:38.443711 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:38.443668 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6wl6d\"" Oct 30 00:10:38.445835 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:38.445812 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-c5jln"] Oct 30 00:10:38.566998 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:38.566958 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nbls\" (UniqueName: \"kubernetes.io/projected/b2e1b1cc-ac20-4ebd-a784-05986577b551-kube-api-access-2nbls\") pod \"authorino-57f7f8659f-c5jln\" (UID: \"b2e1b1cc-ac20-4ebd-a784-05986577b551\") " pod="kuadrant-system/authorino-57f7f8659f-c5jln" Oct 30 00:10:38.668298 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:38.668260 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nbls\" (UniqueName: \"kubernetes.io/projected/b2e1b1cc-ac20-4ebd-a784-05986577b551-kube-api-access-2nbls\") pod \"authorino-57f7f8659f-c5jln\" (UID: \"b2e1b1cc-ac20-4ebd-a784-05986577b551\") " pod="kuadrant-system/authorino-57f7f8659f-c5jln" Oct 30 00:10:38.688876 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:38.688845 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nbls\" (UniqueName: \"kubernetes.io/projected/b2e1b1cc-ac20-4ebd-a784-05986577b551-kube-api-access-2nbls\") pod \"authorino-57f7f8659f-c5jln\" (UID: \"b2e1b1cc-ac20-4ebd-a784-05986577b551\") " pod="kuadrant-system/authorino-57f7f8659f-c5jln" Oct 30 00:10:38.741814 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:38.741775 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-c5jln" Oct 30 00:10:39.040146 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:39.040076 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-c5jln"] Oct 30 00:10:39.046080 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:10:39.044494 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2e1b1cc_ac20_4ebd_a784_05986577b551.slice/crio-dbe723e39014213d1fc889d5264de5c72deb8f6e7eacaa70ebb476ef137b72eb WatchSource:0}: Error finding container dbe723e39014213d1fc889d5264de5c72deb8f6e7eacaa70ebb476ef137b72eb: Status 404 returned error can't find the container with id dbe723e39014213d1fc889d5264de5c72deb8f6e7eacaa70ebb476ef137b72eb Oct 30 00:10:39.214495 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:39.214455 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-z72x7" event={"ID":"449bf72c-2ac6-42f7-8174-1e51e4c792cc","Type":"ContainerStarted","Data":"c568d47d118db3754bf2b2bd10930833cf031159ea8202e3d46e5461f563d716"} Oct 30 00:10:39.216188 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:39.216159 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-4shsq" event={"ID":"b0ae782c-47d7-4895-9378-bbab2fc5ecfa","Type":"ContainerStarted","Data":"bd4dafede53d813a64e4af12f1f507be8643efb305fc6a46a9dda065fe8c706a"} Oct 30 00:10:39.219418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:39.219358 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-c5jln" event={"ID":"b2e1b1cc-ac20-4ebd-a784-05986577b551","Type":"ContainerStarted","Data":"dbe723e39014213d1fc889d5264de5c72deb8f6e7eacaa70ebb476ef137b72eb"} Oct 30 00:10:40.241519 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:40.241481 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-4shsq" event={"ID":"b0ae782c-47d7-4895-9378-bbab2fc5ecfa","Type":"ContainerStarted","Data":"74cb3932f276fab2fc9751b25c59d83f38f9465d431c839daadcd9065dbbf2de"} Oct 30 00:10:40.242192 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:40.242171 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-4shsq" Oct 30 00:10:40.281976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:40.281467 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-4shsq" podStartSLOduration=2.096985547 podStartE2EDuration="3.281447858s" totalImagesPullingTime="1.184462311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:10:40.277191022 +0000 UTC m=+355.482692204" watchObservedRunningTime="2026-10-30 00:10:40.281447858 +0000 UTC m=+355.486949040" Oct 30 00:10:41.956299 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:41.956260 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-c5jln"] Oct 30 00:10:42.256942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:42.256851 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-c5jln" event={"ID":"b2e1b1cc-ac20-4ebd-a784-05986577b551","Type":"ContainerStarted","Data":"b992fe416aae9cc7cffb9a682c7863e3f8e543cc0614c44c47379aa351007d8c"} Oct 30 00:10:42.292305 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:42.292242 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-c5jln" podStartSLOduration=1.9602637889999999 podStartE2EDuration="4.292223804s" totalImagesPullingTime="2.331960015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:10:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:10:42.290143212 +0000 UTC m=+357.495644396" watchObservedRunningTime="2026-10-30 00:10:42.292223804 +0000 UTC m=+357.497724985" Oct 30 00:10:43.263517 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:43.263473 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-c5jln" podUID="b2e1b1cc-ac20-4ebd-a784-05986577b551" containerName="authorino" containerID="cri-o://b992fe416aae9cc7cffb9a682c7863e3f8e543cc0614c44c47379aa351007d8c" gracePeriod=30 Oct 30 00:10:43.509512 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:43.509483 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-c5jln" Oct 30 00:10:43.613780 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:43.613700 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2e1b1cc-ac20-4ebd-a784-05986577b551-kube-api-access-2nbls\" (UniqueName: \"kubernetes.io/projected/b2e1b1cc-ac20-4ebd-a784-05986577b551-kube-api-access-2nbls\") pod \"b2e1b1cc-ac20-4ebd-a784-05986577b551\" (UID: \"b2e1b1cc-ac20-4ebd-a784-05986577b551\") " Oct 30 00:10:43.615717 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:43.615690 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e1b1cc-ac20-4ebd-a784-05986577b551-kube-api-access-2nbls" pod "b2e1b1cc-ac20-4ebd-a784-05986577b551" (UID: "b2e1b1cc-ac20-4ebd-a784-05986577b551"). InnerVolumeSpecName "kube-api-access-2nbls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:10:43.715168 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:43.715132 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nbls\" (UniqueName: \"kubernetes.io/projected/b2e1b1cc-ac20-4ebd-a784-05986577b551-kube-api-access-2nbls\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:10:44.275519 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:44.275475 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-z72x7" event={"ID":"449bf72c-2ac6-42f7-8174-1e51e4c792cc","Type":"ContainerStarted","Data":"7eb4fbd8d5ea5168eec3687efcaf1498e2c6357ab6075c5a1b10837341d3967d"} Oct 30 00:10:44.277096 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:44.277067 5001 generic.go:356] "Generic (PLEG): container finished" podID="b2e1b1cc-ac20-4ebd-a784-05986577b551" containerID="b992fe416aae9cc7cffb9a682c7863e3f8e543cc0614c44c47379aa351007d8c" exitCode=0 Oct 30 00:10:44.277201 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:44.277104 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-c5jln" event={"ID":"b2e1b1cc-ac20-4ebd-a784-05986577b551","Type":"ContainerDied","Data":"b992fe416aae9cc7cffb9a682c7863e3f8e543cc0614c44c47379aa351007d8c"} Oct 30 00:10:44.277201 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:44.277116 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-c5jln" Oct 30 00:10:44.277201 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:44.277127 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-c5jln" event={"ID":"b2e1b1cc-ac20-4ebd-a784-05986577b551","Type":"ContainerDied","Data":"dbe723e39014213d1fc889d5264de5c72deb8f6e7eacaa70ebb476ef137b72eb"} Oct 30 00:10:44.277201 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:44.277144 5001 scope.go:122] "RemoveContainer" containerID="b992fe416aae9cc7cffb9a682c7863e3f8e543cc0614c44c47379aa351007d8c" Oct 30 00:10:44.287051 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:44.287033 5001 scope.go:122] "RemoveContainer" containerID="b992fe416aae9cc7cffb9a682c7863e3f8e543cc0614c44c47379aa351007d8c" Oct 30 00:10:44.287302 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:10:44.287282 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b992fe416aae9cc7cffb9a682c7863e3f8e543cc0614c44c47379aa351007d8c\": container with ID starting with b992fe416aae9cc7cffb9a682c7863e3f8e543cc0614c44c47379aa351007d8c not found: ID does not exist" containerID="b992fe416aae9cc7cffb9a682c7863e3f8e543cc0614c44c47379aa351007d8c" Oct 30 00:10:44.287366 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:44.287317 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b992fe416aae9cc7cffb9a682c7863e3f8e543cc0614c44c47379aa351007d8c"} err="failed to get container status \"b992fe416aae9cc7cffb9a682c7863e3f8e543cc0614c44c47379aa351007d8c\": rpc error: code = NotFound desc = could not find container \"b992fe416aae9cc7cffb9a682c7863e3f8e543cc0614c44c47379aa351007d8c\": container with ID starting with b992fe416aae9cc7cffb9a682c7863e3f8e543cc0614c44c47379aa351007d8c not found: ID does not exist" Oct 30 00:10:44.311429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:44.311377 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-z72x7" podStartSLOduration=2.202090073 podStartE2EDuration="7.311359708s" totalImagesPullingTime="5.109269635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:10:44.310272868 +0000 UTC m=+359.515774060" watchObservedRunningTime="2026-10-30 00:10:44.311359708 +0000 UTC m=+359.516860887" Oct 30 00:10:44.404750 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:44.404718 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-c5jln"] Oct 30 00:10:44.406738 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:44.406711 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-c5jln"] Oct 30 00:10:44.916368 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:44.916339 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2e1b1cc-ac20-4ebd-a784-05986577b551" path="/var/lib/kubelet/pods/b2e1b1cc-ac20-4ebd-a784-05986577b551/volumes" Oct 30 00:10:47.320894 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:10:47.320858 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:10:51.965688 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:51.965652 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4shsq"] Oct 30 00:10:51.966170 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:51.965877 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-4shsq" podUID="b0ae782c-47d7-4895-9378-bbab2fc5ecfa" containerName="limitador" containerID="cri-o://74cb3932f276fab2fc9751b25c59d83f38f9465d431c839daadcd9065dbbf2de" gracePeriod=30 Oct 30 00:10:51.966629 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:51.966609 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-4shsq" Oct 30 00:10:52.326199 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.326170 5001 generic.go:356] "Generic (PLEG): container finished" podID="b0ae782c-47d7-4895-9378-bbab2fc5ecfa" containerID="74cb3932f276fab2fc9751b25c59d83f38f9465d431c839daadcd9065dbbf2de" exitCode=0 Oct 30 00:10:52.326364 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.326221 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-4shsq" event={"ID":"b0ae782c-47d7-4895-9378-bbab2fc5ecfa","Type":"ContainerDied","Data":"74cb3932f276fab2fc9751b25c59d83f38f9465d431c839daadcd9065dbbf2de"} Oct 30 00:10:52.460472 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.460435 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm"] Oct 30 00:10:52.460878 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.460864 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b2e1b1cc-ac20-4ebd-a784-05986577b551" containerName="authorino" Oct 30 00:10:52.460943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.460879 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2e1b1cc-ac20-4ebd-a784-05986577b551" containerName="authorino" Oct 30 00:10:52.460993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.460980 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2e1b1cc-ac20-4ebd-a784-05986577b551" containerName="authorino" Oct 30 00:10:52.464060 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.464039 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" Oct 30 00:10:52.477850 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.477825 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 00:10:52.477850 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.477836 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 00:10:52.478018 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.477907 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 00:10:52.478528 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.478507 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-nht5w\"" Oct 30 00:10:52.491366 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.491338 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm"] Oct 30 00:10:52.593083 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.593009 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d50b1c24-43df-4ca1-b869-5e1d0d9d67ce-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z4vm\" (UID: \"d50b1c24-43df-4ca1-b869-5e1d0d9d67ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" Oct 30 00:10:52.593083 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.593043 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d50b1c24-43df-4ca1-b869-5e1d0d9d67ce-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z4vm\" (UID: \"d50b1c24-43df-4ca1-b869-5e1d0d9d67ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" Oct 30 00:10:52.593083 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.593078 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d50b1c24-43df-4ca1-b869-5e1d0d9d67ce-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z4vm\" (UID: \"d50b1c24-43df-4ca1-b869-5e1d0d9d67ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" Oct 30 00:10:52.593290 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.593190 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d50b1c24-43df-4ca1-b869-5e1d0d9d67ce-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z4vm\" (UID: \"d50b1c24-43df-4ca1-b869-5e1d0d9d67ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" Oct 30 00:10:52.593290 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.593275 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d50b1c24-43df-4ca1-b869-5e1d0d9d67ce-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z4vm\" (UID: \"d50b1c24-43df-4ca1-b869-5e1d0d9d67ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" Oct 30 00:10:52.593356 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.593307 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d50b1c24-43df-4ca1-b869-5e1d0d9d67ce-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z4vm\" (UID: \"d50b1c24-43df-4ca1-b869-5e1d0d9d67ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" Oct 30 00:10:52.593356 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.593343 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-925nc\" (UniqueName: \"kubernetes.io/projected/d50b1c24-43df-4ca1-b869-5e1d0d9d67ce-kube-api-access-925nc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z4vm\" (UID: \"d50b1c24-43df-4ca1-b869-5e1d0d9d67ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" Oct 30 00:10:52.694454 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.694411 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-925nc\" (UniqueName: \"kubernetes.io/projected/d50b1c24-43df-4ca1-b869-5e1d0d9d67ce-kube-api-access-925nc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z4vm\" (UID: \"d50b1c24-43df-4ca1-b869-5e1d0d9d67ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" Oct 30 00:10:52.694623 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.694492 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d50b1c24-43df-4ca1-b869-5e1d0d9d67ce-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z4vm\" (UID: \"d50b1c24-43df-4ca1-b869-5e1d0d9d67ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" Oct 30 00:10:52.694623 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.694509 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d50b1c24-43df-4ca1-b869-5e1d0d9d67ce-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z4vm\" (UID: \"d50b1c24-43df-4ca1-b869-5e1d0d9d67ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" Oct 30 00:10:52.694623 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.694529 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d50b1c24-43df-4ca1-b869-5e1d0d9d67ce-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z4vm\" (UID: \"d50b1c24-43df-4ca1-b869-5e1d0d9d67ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" Oct 30 00:10:52.694623 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.694546 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d50b1c24-43df-4ca1-b869-5e1d0d9d67ce-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z4vm\" (UID: \"d50b1c24-43df-4ca1-b869-5e1d0d9d67ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" Oct 30 00:10:52.694623 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.694577 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d50b1c24-43df-4ca1-b869-5e1d0d9d67ce-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z4vm\" (UID: \"d50b1c24-43df-4ca1-b869-5e1d0d9d67ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" Oct 30 00:10:52.694623 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.694607 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d50b1c24-43df-4ca1-b869-5e1d0d9d67ce-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z4vm\" (UID: \"d50b1c24-43df-4ca1-b869-5e1d0d9d67ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" Oct 30 00:10:52.696953 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.695887 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d50b1c24-43df-4ca1-b869-5e1d0d9d67ce-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z4vm\" (UID: \"d50b1c24-43df-4ca1-b869-5e1d0d9d67ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" Oct 30 00:10:52.698061 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.698010 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d50b1c24-43df-4ca1-b869-5e1d0d9d67ce-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z4vm\" (UID: \"d50b1c24-43df-4ca1-b869-5e1d0d9d67ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" Oct 30 00:10:52.698189 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.698093 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d50b1c24-43df-4ca1-b869-5e1d0d9d67ce-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z4vm\" (UID: \"d50b1c24-43df-4ca1-b869-5e1d0d9d67ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" Oct 30 00:10:52.698189 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.698166 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d50b1c24-43df-4ca1-b869-5e1d0d9d67ce-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z4vm\" (UID: \"d50b1c24-43df-4ca1-b869-5e1d0d9d67ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" Oct 30 00:10:52.704639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.704615 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d50b1c24-43df-4ca1-b869-5e1d0d9d67ce-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z4vm\" (UID: \"d50b1c24-43df-4ca1-b869-5e1d0d9d67ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" Oct 30 00:10:52.715217 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.715192 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d50b1c24-43df-4ca1-b869-5e1d0d9d67ce-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z4vm\" (UID: \"d50b1c24-43df-4ca1-b869-5e1d0d9d67ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" Oct 30 00:10:52.715478 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.715452 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-925nc\" (UniqueName: \"kubernetes.io/projected/d50b1c24-43df-4ca1-b869-5e1d0d9d67ce-kube-api-access-925nc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2z4vm\" (UID: \"d50b1c24-43df-4ca1-b869-5e1d0d9d67ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" Oct 30 00:10:52.776070 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.776026 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" Oct 30 00:10:52.937586 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:52.937560 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-4shsq" Oct 30 00:10:53.088202 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:53.088170 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm"] Oct 30 00:10:53.091834 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:10:53.091795 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50b1c24_43df_4ca1_b869_5e1d0d9d67ce.slice/crio-73e9ddb95d9b01db9589105b5ee66fcb0dab1994679cce6928423eb039a1858e WatchSource:0}: Error finding container 73e9ddb95d9b01db9589105b5ee66fcb0dab1994679cce6928423eb039a1858e: Status 404 returned error can't find the container with id 73e9ddb95d9b01db9589105b5ee66fcb0dab1994679cce6928423eb039a1858e Oct 30 00:10:53.098242 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:53.098220 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0ae782c-47d7-4895-9378-bbab2fc5ecfa-config-file\" (UniqueName: \"kubernetes.io/configmap/b0ae782c-47d7-4895-9378-bbab2fc5ecfa-config-file\") pod \"b0ae782c-47d7-4895-9378-bbab2fc5ecfa\" (UID: \"b0ae782c-47d7-4895-9378-bbab2fc5ecfa\") " Oct 30 00:10:53.098335 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:53.098264 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0ae782c-47d7-4895-9378-bbab2fc5ecfa-kube-api-access-kz68d\" (UniqueName: \"kubernetes.io/projected/b0ae782c-47d7-4895-9378-bbab2fc5ecfa-kube-api-access-kz68d\") pod \"b0ae782c-47d7-4895-9378-bbab2fc5ecfa\" (UID: \"b0ae782c-47d7-4895-9378-bbab2fc5ecfa\") " Oct 30 00:10:53.098595 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:53.098571 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0ae782c-47d7-4895-9378-bbab2fc5ecfa-config-file" pod "b0ae782c-47d7-4895-9378-bbab2fc5ecfa" (UID: "b0ae782c-47d7-4895-9378-bbab2fc5ecfa"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:10:53.100330 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:53.100298 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ae782c-47d7-4895-9378-bbab2fc5ecfa-kube-api-access-kz68d" pod "b0ae782c-47d7-4895-9378-bbab2fc5ecfa" (UID: "b0ae782c-47d7-4895-9378-bbab2fc5ecfa"). InnerVolumeSpecName "kube-api-access-kz68d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:10:53.199821 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:53.199747 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz68d\" (UniqueName: \"kubernetes.io/projected/b0ae782c-47d7-4895-9378-bbab2fc5ecfa-kube-api-access-kz68d\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:10:53.199821 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:53.199774 5001 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b0ae782c-47d7-4895-9378-bbab2fc5ecfa-config-file\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:10:53.332893 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:53.332859 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-4shsq" Oct 30 00:10:53.333079 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:53.332893 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-4shsq" event={"ID":"b0ae782c-47d7-4895-9378-bbab2fc5ecfa","Type":"ContainerDied","Data":"bd4dafede53d813a64e4af12f1f507be8643efb305fc6a46a9dda065fe8c706a"} Oct 30 00:10:53.333079 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:53.332971 5001 scope.go:122] "RemoveContainer" containerID="74cb3932f276fab2fc9751b25c59d83f38f9465d431c839daadcd9065dbbf2de" Oct 30 00:10:53.334344 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:53.334314 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" event={"ID":"d50b1c24-43df-4ca1-b869-5e1d0d9d67ce","Type":"ContainerStarted","Data":"73e9ddb95d9b01db9589105b5ee66fcb0dab1994679cce6928423eb039a1858e"} Oct 30 00:10:53.377021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:53.376998 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4shsq"] Oct 30 00:10:53.380906 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:53.380884 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4shsq"] Oct 30 00:10:53.861353 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:53.861320 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-6rk8f"] Oct 30 00:10:53.861882 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:53.861856 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b0ae782c-47d7-4895-9378-bbab2fc5ecfa" containerName="limitador" Oct 30 00:10:53.861882 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:53.861879 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ae782c-47d7-4895-9378-bbab2fc5ecfa" containerName="limitador" Oct 30 00:10:53.862074 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:53.862032 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ae782c-47d7-4895-9378-bbab2fc5ecfa" containerName="limitador" Oct 30 00:10:53.866794 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:53.866776 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6rk8f" Oct 30 00:10:53.885558 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:53.885523 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-6rk8f"] Oct 30 00:10:53.921563 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:53.921526 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-kwjns\"" Oct 30 00:10:53.921563 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:53.921553 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 00:10:54.008631 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:54.008594 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99f048c8-2319-46ef-87a5-b3cf06ba0332-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-6rk8f\" (UID: \"99f048c8-2319-46ef-87a5-b3cf06ba0332\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6rk8f" Oct 30 00:10:54.008817 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:54.008664 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8lsm\" (UniqueName: \"kubernetes.io/projected/99f048c8-2319-46ef-87a5-b3cf06ba0332-kube-api-access-j8lsm\") pod \"llmisvc-controller-manager-869d5dcc5d-6rk8f\" (UID: \"99f048c8-2319-46ef-87a5-b3cf06ba0332\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6rk8f" Oct 30 00:10:54.110037 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:54.110005 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99f048c8-2319-46ef-87a5-b3cf06ba0332-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-6rk8f\" (UID: \"99f048c8-2319-46ef-87a5-b3cf06ba0332\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6rk8f" Oct 30 00:10:54.110476 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:54.110057 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8lsm\" (UniqueName: \"kubernetes.io/projected/99f048c8-2319-46ef-87a5-b3cf06ba0332-kube-api-access-j8lsm\") pod \"llmisvc-controller-manager-869d5dcc5d-6rk8f\" (UID: \"99f048c8-2319-46ef-87a5-b3cf06ba0332\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6rk8f" Oct 30 00:10:54.113068 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:54.112986 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99f048c8-2319-46ef-87a5-b3cf06ba0332-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-6rk8f\" (UID: \"99f048c8-2319-46ef-87a5-b3cf06ba0332\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6rk8f" Oct 30 00:10:54.131288 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:54.131260 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8lsm\" (UniqueName: \"kubernetes.io/projected/99f048c8-2319-46ef-87a5-b3cf06ba0332-kube-api-access-j8lsm\") pod \"llmisvc-controller-manager-869d5dcc5d-6rk8f\" (UID: \"99f048c8-2319-46ef-87a5-b3cf06ba0332\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6rk8f" Oct 30 00:10:54.182781 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:54.182751 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6rk8f" Oct 30 00:10:54.485252 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:54.485167 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-6rk8f"] Oct 30 00:10:54.489495 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:10:54.489461 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod99f048c8_2319_46ef_87a5_b3cf06ba0332.slice/crio-c689240a6d1e2fa473e0ee20c8203378ab01ab6987db66966d57d7e294b1d217 WatchSource:0}: Error finding container c689240a6d1e2fa473e0ee20c8203378ab01ab6987db66966d57d7e294b1d217: Status 404 returned error can't find the container with id c689240a6d1e2fa473e0ee20c8203378ab01ab6987db66966d57d7e294b1d217 Oct 30 00:10:54.917016 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:54.916987 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0ae782c-47d7-4895-9378-bbab2fc5ecfa" path="/var/lib/kubelet/pods/b0ae782c-47d7-4895-9378-bbab2fc5ecfa/volumes" Oct 30 00:10:55.353897 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:55.353855 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6rk8f" event={"ID":"99f048c8-2319-46ef-87a5-b3cf06ba0332","Type":"ContainerStarted","Data":"c689240a6d1e2fa473e0ee20c8203378ab01ab6987db66966d57d7e294b1d217"} Oct 30 00:10:55.632604 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:55.632564 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:10:55.632706 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:55.632649 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:10:56.362862 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:56.362825 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" event={"ID":"d50b1c24-43df-4ca1-b869-5e1d0d9d67ce","Type":"ContainerStarted","Data":"01a9dd7eecaab38e0fa624b7039a977439b2c4631fe839b9a14542329c833bc5"} Oct 30 00:10:56.363407 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:56.363006 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" Oct 30 00:10:56.364936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:56.364888 5001 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-2z4vm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:10:56.365061 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:56.364950 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" podUID="d50b1c24-43df-4ca1-b869-5e1d0d9d67ce" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:10:56.428259 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:56.428196 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" podStartSLOduration=1.89040411 podStartE2EDuration="4.428177518s" totalImagesPullingTime="2.537773408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:10:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:10:56.425738435 +0000 UTC m=+371.631239640" watchObservedRunningTime="2026-10-30 00:10:56.428177518 +0000 UTC m=+371.633678700" Oct 30 00:10:57.369948 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:57.369901 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6rk8f" event={"ID":"99f048c8-2319-46ef-87a5-b3cf06ba0332","Type":"ContainerStarted","Data":"a08fcef4ea30f2cf5d4e248af2c2ce7bd5c61bf50be583c7838e60519bb99826"} Oct 30 00:10:57.370444 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:57.370072 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6rk8f" Oct 30 00:10:57.370722 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:57.370702 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" Oct 30 00:10:57.407008 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:10:57.406966 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6rk8f" podStartSLOduration=2.455514184 podStartE2EDuration="4.40695258s" totalImagesPullingTime="1.951438396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:10:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:10:57.405668674 +0000 UTC m=+372.611169854" watchObservedRunningTime="2026-10-30 00:10:57.40695258 +0000 UTC m=+372.612453758" Oct 30 00:11:17.321781 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:11:17.321745 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:11:28.378240 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:11:28.378159 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6rk8f" Oct 30 00:11:47.321654 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:11:47.321611 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:12:01.726469 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:01.726370 5001 generic.go:356] "Generic (PLEG): container finished" podID="35bc1599-694f-4f89-9a8a-7667310b9b90" containerID="f729a1e075e62ccca8b57280d27c2a18975fbeaf902c4c76fae53e6e51fd0d2b" exitCode=0 Oct 30 00:12:01.726469 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:01.726446 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" event={"ID":"35bc1599-694f-4f89-9a8a-7667310b9b90","Type":"ContainerDied","Data":"f729a1e075e62ccca8b57280d27c2a18975fbeaf902c4c76fae53e6e51fd0d2b"} Oct 30 00:12:01.726959 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:01.726816 5001 scope.go:122] "RemoveContainer" containerID="f729a1e075e62ccca8b57280d27c2a18975fbeaf902c4c76fae53e6e51fd0d2b" Oct 30 00:12:01.727978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:01.727946 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:12:01.728035 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:01.728023 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:12:01.728088 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:01.728057 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:12:02.146691 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:02.146652 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:12:02.189753 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:02.189721 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-p564d"] Oct 30 00:12:02.192452 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:02.192436 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-p564d" Oct 30 00:12:02.229934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:02.214758 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-p564d"] Oct 30 00:12:02.240836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:02.240796 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfzdg\" (UniqueName: \"kubernetes.io/projected/f408e652-4069-4d1d-b5b0-abc34593794c-kube-api-access-cfzdg\") pod \"llmisvc-controller-manager-6466c554d8-p564d\" (UID: \"f408e652-4069-4d1d-b5b0-abc34593794c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-p564d" Oct 30 00:12:02.241022 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:02.240851 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f408e652-4069-4d1d-b5b0-abc34593794c-cert\") pod \"llmisvc-controller-manager-6466c554d8-p564d\" (UID: \"f408e652-4069-4d1d-b5b0-abc34593794c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-p564d" Oct 30 00:12:02.341474 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:02.341441 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f408e652-4069-4d1d-b5b0-abc34593794c-cert\") pod \"llmisvc-controller-manager-6466c554d8-p564d\" (UID: \"f408e652-4069-4d1d-b5b0-abc34593794c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-p564d" Oct 30 00:12:02.341640 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:02.341550 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfzdg\" (UniqueName: \"kubernetes.io/projected/f408e652-4069-4d1d-b5b0-abc34593794c-kube-api-access-cfzdg\") pod \"llmisvc-controller-manager-6466c554d8-p564d\" (UID: \"f408e652-4069-4d1d-b5b0-abc34593794c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-p564d" Oct 30 00:12:02.345490 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:02.345457 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f408e652-4069-4d1d-b5b0-abc34593794c-cert\") pod \"llmisvc-controller-manager-6466c554d8-p564d\" (UID: \"f408e652-4069-4d1d-b5b0-abc34593794c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-p564d" Oct 30 00:12:02.362228 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:02.362205 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfzdg\" (UniqueName: \"kubernetes.io/projected/f408e652-4069-4d1d-b5b0-abc34593794c-kube-api-access-cfzdg\") pod \"llmisvc-controller-manager-6466c554d8-p564d\" (UID: \"f408e652-4069-4d1d-b5b0-abc34593794c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-p564d" Oct 30 00:12:02.504170 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:02.504085 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-p564d" Oct 30 00:12:02.738871 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:02.738842 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" event={"ID":"35bc1599-694f-4f89-9a8a-7667310b9b90","Type":"ContainerStarted","Data":"f9ca5bf35f7104390a99ff08f661910cc9f0e185ae680b4d385dc30643eef04f"} Oct 30 00:12:02.833504 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:02.833270 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-p564d"] Oct 30 00:12:03.237958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:03.237891 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:12:03.239391 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:03.239365 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 30 00:12:03.239455 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:03.239429 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" podUID="35bc1599-694f-4f89-9a8a-7667310b9b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 30 00:12:03.745781 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:03.745744 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-p564d" event={"ID":"f408e652-4069-4d1d-b5b0-abc34593794c","Type":"ContainerStarted","Data":"42453b51ac78ceed0ccbe76c432b910f2d4e981aad16677842f5dd7f7d283e4a"} Oct 30 00:12:03.746287 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:03.745800 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-p564d" event={"ID":"f408e652-4069-4d1d-b5b0-abc34593794c","Type":"ContainerStarted","Data":"26882fe5043996de3e43726c95a8de2c0fd1d3e66ee3bc757f9ae0aa2c727f39"} Oct 30 00:12:03.746287 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:03.746049 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-p564d" Oct 30 00:12:03.782086 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:03.782035 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-p564d" podStartSLOduration=1.474218004 podStartE2EDuration="1.782022323s" totalImagesPullingTime="307.804319ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:12:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:12:03.780326514 +0000 UTC m=+438.985827695" watchObservedRunningTime="2026-10-30 00:12:03.782022323 +0000 UTC m=+438.987523501" Oct 30 00:12:04.237276 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:04.237242 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 30 00:12:04.237460 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:04.237298 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" podUID="35bc1599-694f-4f89-9a8a-7667310b9b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 30 00:12:05.237353 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:05.237313 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 30 00:12:05.237777 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:05.237372 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" podUID="35bc1599-694f-4f89-9a8a-7667310b9b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 30 00:12:06.237431 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:06.237394 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 30 00:12:06.237861 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:06.237450 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" podUID="35bc1599-694f-4f89-9a8a-7667310b9b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 30 00:12:07.237377 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:07.237340 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 30 00:12:07.237556 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:07.237398 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" podUID="35bc1599-694f-4f89-9a8a-7667310b9b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 30 00:12:08.241646 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:08.241613 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:12:08.242063 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:08.241890 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:12:08.242649 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:08.242629 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:12:17.322053 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:12:17.322009 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:12:34.754069 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:34.754035 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-p564d" Oct 30 00:12:34.818703 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:34.818666 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-6rk8f"] Oct 30 00:12:34.818972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:34.818947 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6rk8f" podUID="99f048c8-2319-46ef-87a5-b3cf06ba0332" containerName="manager" containerID="cri-o://a08fcef4ea30f2cf5d4e248af2c2ce7bd5c61bf50be583c7838e60519bb99826" gracePeriod=30 Oct 30 00:12:35.066947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:35.066923 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6rk8f" Oct 30 00:12:35.235447 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:35.235412 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99f048c8-2319-46ef-87a5-b3cf06ba0332-kube-api-access-j8lsm\" (UniqueName: \"kubernetes.io/projected/99f048c8-2319-46ef-87a5-b3cf06ba0332-kube-api-access-j8lsm\") pod \"99f048c8-2319-46ef-87a5-b3cf06ba0332\" (UID: \"99f048c8-2319-46ef-87a5-b3cf06ba0332\") " Oct 30 00:12:35.235612 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:35.235535 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99f048c8-2319-46ef-87a5-b3cf06ba0332-cert\" (UniqueName: \"kubernetes.io/secret/99f048c8-2319-46ef-87a5-b3cf06ba0332-cert\") pod \"99f048c8-2319-46ef-87a5-b3cf06ba0332\" (UID: \"99f048c8-2319-46ef-87a5-b3cf06ba0332\") " Oct 30 00:12:35.237646 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:35.237615 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99f048c8-2319-46ef-87a5-b3cf06ba0332-kube-api-access-j8lsm" pod "99f048c8-2319-46ef-87a5-b3cf06ba0332" (UID: "99f048c8-2319-46ef-87a5-b3cf06ba0332"). InnerVolumeSpecName "kube-api-access-j8lsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:12:35.237761 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:35.237693 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99f048c8-2319-46ef-87a5-b3cf06ba0332-cert" pod "99f048c8-2319-46ef-87a5-b3cf06ba0332" (UID: "99f048c8-2319-46ef-87a5-b3cf06ba0332"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:12:35.336895 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:35.336812 5001 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99f048c8-2319-46ef-87a5-b3cf06ba0332-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:12:35.336895 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:35.336844 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8lsm\" (UniqueName: \"kubernetes.io/projected/99f048c8-2319-46ef-87a5-b3cf06ba0332-kube-api-access-j8lsm\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:12:35.916582 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:35.916549 5001 generic.go:356] "Generic (PLEG): container finished" podID="99f048c8-2319-46ef-87a5-b3cf06ba0332" containerID="a08fcef4ea30f2cf5d4e248af2c2ce7bd5c61bf50be583c7838e60519bb99826" exitCode=0 Oct 30 00:12:35.917024 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:35.916587 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6rk8f" Oct 30 00:12:35.917024 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:35.916633 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6rk8f" event={"ID":"99f048c8-2319-46ef-87a5-b3cf06ba0332","Type":"ContainerDied","Data":"a08fcef4ea30f2cf5d4e248af2c2ce7bd5c61bf50be583c7838e60519bb99826"} Oct 30 00:12:35.917024 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:35.916682 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6rk8f" event={"ID":"99f048c8-2319-46ef-87a5-b3cf06ba0332","Type":"ContainerDied","Data":"c689240a6d1e2fa473e0ee20c8203378ab01ab6987db66966d57d7e294b1d217"} Oct 30 00:12:35.917024 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:35.916705 5001 scope.go:122] "RemoveContainer" containerID="a08fcef4ea30f2cf5d4e248af2c2ce7bd5c61bf50be583c7838e60519bb99826" Oct 30 00:12:35.925982 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:35.925963 5001 scope.go:122] "RemoveContainer" containerID="a08fcef4ea30f2cf5d4e248af2c2ce7bd5c61bf50be583c7838e60519bb99826" Oct 30 00:12:35.926237 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:12:35.926218 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a08fcef4ea30f2cf5d4e248af2c2ce7bd5c61bf50be583c7838e60519bb99826\": container with ID starting with a08fcef4ea30f2cf5d4e248af2c2ce7bd5c61bf50be583c7838e60519bb99826 not found: ID does not exist" containerID="a08fcef4ea30f2cf5d4e248af2c2ce7bd5c61bf50be583c7838e60519bb99826" Oct 30 00:12:35.926328 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:35.926246 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a08fcef4ea30f2cf5d4e248af2c2ce7bd5c61bf50be583c7838e60519bb99826"} err="failed to get container status \"a08fcef4ea30f2cf5d4e248af2c2ce7bd5c61bf50be583c7838e60519bb99826\": rpc error: code = NotFound desc = could not find container \"a08fcef4ea30f2cf5d4e248af2c2ce7bd5c61bf50be583c7838e60519bb99826\": container with ID starting with a08fcef4ea30f2cf5d4e248af2c2ce7bd5c61bf50be583c7838e60519bb99826 not found: ID does not exist" Oct 30 00:12:35.960374 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:35.960344 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-6rk8f"] Oct 30 00:12:35.969986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:35.969964 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-6rk8f"] Oct 30 00:12:36.922646 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:12:36.922612 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99f048c8-2319-46ef-87a5-b3cf06ba0332" path="/var/lib/kubelet/pods/99f048c8-2319-46ef-87a5-b3cf06ba0332/volumes" Oct 30 00:12:47.322376 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:12:47.322341 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:13:17.321356 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:13:17.321270 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:13:47.322740 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:13:47.322700 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:13:49.961842 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:49.961806 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p"] Oct 30 00:13:49.962327 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:49.962264 5001 cpu_manager.go:414] "containerMap: removing container" podUID="99f048c8-2319-46ef-87a5-b3cf06ba0332" containerName="manager" Oct 30 00:13:49.962327 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:49.962279 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99f048c8-2319-46ef-87a5-b3cf06ba0332" containerName="manager" Oct 30 00:13:49.962400 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:49.962334 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="99f048c8-2319-46ef-87a5-b3cf06ba0332" containerName="manager" Oct 30 00:13:49.964424 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:49.964408 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" Oct 30 00:13:49.994040 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:49.994006 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p"] Oct 30 00:13:50.004657 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:50.004627 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40f44089-5854-4325-820f-b3481352eb28-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ff75p\" (UID: \"40f44089-5854-4325-820f-b3481352eb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" Oct 30 00:13:50.004872 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:50.004848 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwtss\" (UniqueName: \"kubernetes.io/projected/40f44089-5854-4325-820f-b3481352eb28-kube-api-access-mwtss\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ff75p\" (UID: \"40f44089-5854-4325-820f-b3481352eb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" Oct 30 00:13:50.005010 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:50.004883 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40f44089-5854-4325-820f-b3481352eb28-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ff75p\" (UID: \"40f44089-5854-4325-820f-b3481352eb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" Oct 30 00:13:50.005010 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:50.004933 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40f44089-5854-4325-820f-b3481352eb28-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ff75p\" (UID: \"40f44089-5854-4325-820f-b3481352eb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" Oct 30 00:13:50.005010 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:50.004969 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40f44089-5854-4325-820f-b3481352eb28-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ff75p\" (UID: \"40f44089-5854-4325-820f-b3481352eb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" Oct 30 00:13:50.005211 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:50.005188 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f44089-5854-4325-820f-b3481352eb28-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ff75p\" (UID: \"40f44089-5854-4325-820f-b3481352eb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" Oct 30 00:13:50.005276 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:50.005239 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f44089-5854-4325-820f-b3481352eb28-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ff75p\" (UID: \"40f44089-5854-4325-820f-b3481352eb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" Oct 30 00:13:50.005333 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:50.005310 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40f44089-5854-4325-820f-b3481352eb28-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ff75p\" (UID: \"40f44089-5854-4325-820f-b3481352eb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" Oct 30 00:13:50.005386 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:50.005343 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40f44089-5854-4325-820f-b3481352eb28-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ff75p\" (UID: \"40f44089-5854-4325-820f-b3481352eb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" Oct 30 00:13:50.105872 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:50.105835 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwtss\" (UniqueName: \"kubernetes.io/projected/40f44089-5854-4325-820f-b3481352eb28-kube-api-access-mwtss\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ff75p\" (UID: \"40f44089-5854-4325-820f-b3481352eb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" Oct 30 00:13:50.105872 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:50.105873 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40f44089-5854-4325-820f-b3481352eb28-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ff75p\" (UID: \"40f44089-5854-4325-820f-b3481352eb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" Oct 30 00:13:50.106151 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:50.105898 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40f44089-5854-4325-820f-b3481352eb28-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ff75p\" (UID: \"40f44089-5854-4325-820f-b3481352eb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" Oct 30 00:13:50.106151 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:50.105942 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40f44089-5854-4325-820f-b3481352eb28-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ff75p\" (UID: \"40f44089-5854-4325-820f-b3481352eb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" Oct 30 00:13:50.106151 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:50.105964 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f44089-5854-4325-820f-b3481352eb28-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ff75p\" (UID: \"40f44089-5854-4325-820f-b3481352eb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" Oct 30 00:13:50.106151 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:50.105986 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f44089-5854-4325-820f-b3481352eb28-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ff75p\" (UID: \"40f44089-5854-4325-820f-b3481352eb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" Oct 30 00:13:50.106151 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:50.106006 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40f44089-5854-4325-820f-b3481352eb28-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ff75p\" (UID: \"40f44089-5854-4325-820f-b3481352eb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" Oct 30 00:13:50.106151 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:50.106027 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40f44089-5854-4325-820f-b3481352eb28-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ff75p\" (UID: \"40f44089-5854-4325-820f-b3481352eb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" Oct 30 00:13:50.106151 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:50.106153 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40f44089-5854-4325-820f-b3481352eb28-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ff75p\" (UID: \"40f44089-5854-4325-820f-b3481352eb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" Oct 30 00:13:50.106518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:50.106355 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40f44089-5854-4325-820f-b3481352eb28-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ff75p\" (UID: \"40f44089-5854-4325-820f-b3481352eb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" Oct 30 00:13:50.106518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:50.106445 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f44089-5854-4325-820f-b3481352eb28-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ff75p\" (UID: \"40f44089-5854-4325-820f-b3481352eb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" Oct 30 00:13:50.106664 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:50.106644 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f44089-5854-4325-820f-b3481352eb28-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ff75p\" (UID: \"40f44089-5854-4325-820f-b3481352eb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" Oct 30 00:13:50.106751 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:50.106730 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40f44089-5854-4325-820f-b3481352eb28-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ff75p\" (UID: \"40f44089-5854-4325-820f-b3481352eb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" Oct 30 00:13:50.106808 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:50.106786 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40f44089-5854-4325-820f-b3481352eb28-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ff75p\" (UID: \"40f44089-5854-4325-820f-b3481352eb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" Oct 30 00:13:50.108517 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:50.108489 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40f44089-5854-4325-820f-b3481352eb28-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ff75p\" (UID: \"40f44089-5854-4325-820f-b3481352eb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" Oct 30 00:13:50.108632 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:50.108581 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40f44089-5854-4325-820f-b3481352eb28-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ff75p\" (UID: \"40f44089-5854-4325-820f-b3481352eb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" Oct 30 00:13:50.131031 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:50.131005 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwtss\" (UniqueName: \"kubernetes.io/projected/40f44089-5854-4325-820f-b3481352eb28-kube-api-access-mwtss\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ff75p\" (UID: \"40f44089-5854-4325-820f-b3481352eb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" Oct 30 00:13:50.131478 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:50.131457 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40f44089-5854-4325-820f-b3481352eb28-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ff75p\" (UID: \"40f44089-5854-4325-820f-b3481352eb28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" Oct 30 00:13:50.277116 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:50.277090 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" Oct 30 00:13:50.588238 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:50.588149 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p"] Oct 30 00:13:50.592099 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:13:50.592076 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40f44089_5854_4325_820f_b3481352eb28.slice/crio-c0d5f18bc6690f4db085ce125e796f29227973bbacd1560f89145763901a9a67 WatchSource:0}: Error finding container c0d5f18bc6690f4db085ce125e796f29227973bbacd1560f89145763901a9a67: Status 404 returned error can't find the container with id c0d5f18bc6690f4db085ce125e796f29227973bbacd1560f89145763901a9a67 Oct 30 00:13:50.594365 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:50.594332 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:13:50.594469 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:50.594407 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:13:50.594469 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:50.594461 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:13:51.347490 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:51.347454 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" event={"ID":"40f44089-5854-4325-820f-b3481352eb28","Type":"ContainerStarted","Data":"c10ca0883fff182d6af6223addfda2b43f745f1f188f0bdd65c748535cae5af6"} Oct 30 00:13:51.347907 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:51.347507 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" event={"ID":"40f44089-5854-4325-820f-b3481352eb28","Type":"ContainerStarted","Data":"c0d5f18bc6690f4db085ce125e796f29227973bbacd1560f89145763901a9a67"} Oct 30 00:13:51.420936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:51.409063 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" podStartSLOduration=2.40904404 podStartE2EDuration="2.40904404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:13:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:13:51.396524852 +0000 UTC m=+546.602026034" watchObservedRunningTime="2026-10-30 00:13:51.40904404 +0000 UTC m=+546.614545223" Oct 30 00:13:52.277655 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:52.277620 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" Oct 30 00:13:52.279014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:52.278982 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ff75p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 00:13:52.279156 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:52.279029 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" podUID="40f44089-5854-4325-820f-b3481352eb28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 00:13:53.277844 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:53.277808 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ff75p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 00:13:53.278263 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:53.277864 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" podUID="40f44089-5854-4325-820f-b3481352eb28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 00:13:54.278402 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:54.278360 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ff75p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 00:13:54.278902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:54.278420 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" podUID="40f44089-5854-4325-820f-b3481352eb28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 00:13:55.277993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:55.277952 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ff75p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 00:13:55.278152 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:55.278018 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" podUID="40f44089-5854-4325-820f-b3481352eb28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 00:13:56.278097 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:56.278058 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ff75p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 00:13:56.278516 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:56.278111 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" podUID="40f44089-5854-4325-820f-b3481352eb28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 00:13:57.278001 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:57.277961 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ff75p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 00:13:57.278219 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:57.278017 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" podUID="40f44089-5854-4325-820f-b3481352eb28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 00:13:59.051165 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:59.051128 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" Oct 30 00:13:59.051640 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:59.051479 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" Oct 30 00:13:59.052276 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:13:59.052256 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" Oct 30 00:14:17.321550 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:14:17.321513 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:14:33.542803 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:33.542720 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-deda90ae-7b4f645874-jjjnn"] Oct 30 00:14:33.550840 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:33.550814 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-deda90ae-7b4f645874-jjjnn" Oct 30 00:14:33.565010 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:33.564981 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-deda90ae-7b4f645874-jjjnn"] Oct 30 00:14:33.574345 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:33.574324 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-deda90ae\"" Oct 30 00:14:33.574682 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:33.574663 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-deda90ae-dockercfg-xbsqb\"" Oct 30 00:14:33.706775 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:33.706745 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84qx6\" (UniqueName: \"kubernetes.io/projected/70171153-613b-4185-bfaa-1dd277298e44-kube-api-access-84qx6\") pod \"payload-processing-e2e-derive-deda90ae-7b4f645874-jjjnn\" (UID: \"70171153-613b-4185-bfaa-1dd277298e44\") " pod="openshift-ingress/payload-processing-e2e-derive-deda90ae-7b4f645874-jjjnn" Oct 30 00:14:33.706978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:33.706812 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70171153-613b-4185-bfaa-1dd277298e44-plugins-config-volume\") pod \"payload-processing-e2e-derive-deda90ae-7b4f645874-jjjnn\" (UID: \"70171153-613b-4185-bfaa-1dd277298e44\") " pod="openshift-ingress/payload-processing-e2e-derive-deda90ae-7b4f645874-jjjnn" Oct 30 00:14:33.808100 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:33.808013 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84qx6\" (UniqueName: \"kubernetes.io/projected/70171153-613b-4185-bfaa-1dd277298e44-kube-api-access-84qx6\") pod \"payload-processing-e2e-derive-deda90ae-7b4f645874-jjjnn\" (UID: \"70171153-613b-4185-bfaa-1dd277298e44\") " pod="openshift-ingress/payload-processing-e2e-derive-deda90ae-7b4f645874-jjjnn" Oct 30 00:14:33.808100 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:33.808091 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70171153-613b-4185-bfaa-1dd277298e44-plugins-config-volume\") pod \"payload-processing-e2e-derive-deda90ae-7b4f645874-jjjnn\" (UID: \"70171153-613b-4185-bfaa-1dd277298e44\") " pod="openshift-ingress/payload-processing-e2e-derive-deda90ae-7b4f645874-jjjnn" Oct 30 00:14:33.808779 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:33.808759 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70171153-613b-4185-bfaa-1dd277298e44-plugins-config-volume\") pod \"payload-processing-e2e-derive-deda90ae-7b4f645874-jjjnn\" (UID: \"70171153-613b-4185-bfaa-1dd277298e44\") " pod="openshift-ingress/payload-processing-e2e-derive-deda90ae-7b4f645874-jjjnn" Oct 30 00:14:33.828970 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:33.828940 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84qx6\" (UniqueName: \"kubernetes.io/projected/70171153-613b-4185-bfaa-1dd277298e44-kube-api-access-84qx6\") pod \"payload-processing-e2e-derive-deda90ae-7b4f645874-jjjnn\" (UID: \"70171153-613b-4185-bfaa-1dd277298e44\") " pod="openshift-ingress/payload-processing-e2e-derive-deda90ae-7b4f645874-jjjnn" Oct 30 00:14:33.874584 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:33.874557 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-deda90ae-7b4f645874-jjjnn" Oct 30 00:14:34.172722 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:34.172641 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-deda90ae-7b4f645874-jjjnn"] Oct 30 00:14:34.175876 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:14:34.175849 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70171153_613b_4185_bfaa_1dd277298e44.slice/crio-109730bd7e784391b8e9170155dfbc8fa95605e7fd691d4abd1531795ecb9de6 WatchSource:0}: Error finding container 109730bd7e784391b8e9170155dfbc8fa95605e7fd691d4abd1531795ecb9de6: Status 404 returned error can't find the container with id 109730bd7e784391b8e9170155dfbc8fa95605e7fd691d4abd1531795ecb9de6 Oct 30 00:14:34.584303 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:34.584256 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-deda90ae-7b4f645874-jjjnn" event={"ID":"70171153-613b-4185-bfaa-1dd277298e44","Type":"ContainerStarted","Data":"109730bd7e784391b8e9170155dfbc8fa95605e7fd691d4abd1531795ecb9de6"} Oct 30 00:14:36.083483 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:36.082646 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-185595da-6f9d6ddd86-6cfxr"] Oct 30 00:14:36.087901 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:36.087779 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-185595da-6f9d6ddd86-6cfxr" Oct 30 00:14:36.107429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:36.107401 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-185595da-6f9d6ddd86-6cfxr"] Oct 30 00:14:36.129040 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:36.128986 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-185595da\"" Oct 30 00:14:36.133629 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:36.133580 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-185595da-dockercfg-wv4lp\"" Oct 30 00:14:36.246026 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:36.245992 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76cf1c1-30bf-4823-a7c5-53380f123448-plugins-config-volume\") pod \"payload-processing-e2e-ait-185595da-6f9d6ddd86-6cfxr\" (UID: \"e76cf1c1-30bf-4823-a7c5-53380f123448\") " pod="openshift-ingress/payload-processing-e2e-ait-185595da-6f9d6ddd86-6cfxr" Oct 30 00:14:36.246279 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:36.246103 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnj4f\" (UniqueName: \"kubernetes.io/projected/e76cf1c1-30bf-4823-a7c5-53380f123448-kube-api-access-mnj4f\") pod \"payload-processing-e2e-ait-185595da-6f9d6ddd86-6cfxr\" (UID: \"e76cf1c1-30bf-4823-a7c5-53380f123448\") " pod="openshift-ingress/payload-processing-e2e-ait-185595da-6f9d6ddd86-6cfxr" Oct 30 00:14:36.346748 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:36.346724 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76cf1c1-30bf-4823-a7c5-53380f123448-plugins-config-volume\") pod \"payload-processing-e2e-ait-185595da-6f9d6ddd86-6cfxr\" (UID: \"e76cf1c1-30bf-4823-a7c5-53380f123448\") " pod="openshift-ingress/payload-processing-e2e-ait-185595da-6f9d6ddd86-6cfxr" Oct 30 00:14:36.346837 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:36.346776 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnj4f\" (UniqueName: \"kubernetes.io/projected/e76cf1c1-30bf-4823-a7c5-53380f123448-kube-api-access-mnj4f\") pod \"payload-processing-e2e-ait-185595da-6f9d6ddd86-6cfxr\" (UID: \"e76cf1c1-30bf-4823-a7c5-53380f123448\") " pod="openshift-ingress/payload-processing-e2e-ait-185595da-6f9d6ddd86-6cfxr" Oct 30 00:14:36.347360 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:36.347342 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76cf1c1-30bf-4823-a7c5-53380f123448-plugins-config-volume\") pod \"payload-processing-e2e-ait-185595da-6f9d6ddd86-6cfxr\" (UID: \"e76cf1c1-30bf-4823-a7c5-53380f123448\") " pod="openshift-ingress/payload-processing-e2e-ait-185595da-6f9d6ddd86-6cfxr" Oct 30 00:14:36.371385 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:36.371360 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnj4f\" (UniqueName: \"kubernetes.io/projected/e76cf1c1-30bf-4823-a7c5-53380f123448-kube-api-access-mnj4f\") pod \"payload-processing-e2e-ait-185595da-6f9d6ddd86-6cfxr\" (UID: \"e76cf1c1-30bf-4823-a7c5-53380f123448\") " pod="openshift-ingress/payload-processing-e2e-ait-185595da-6f9d6ddd86-6cfxr" Oct 30 00:14:36.403510 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:36.403474 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-185595da-6f9d6ddd86-6cfxr" Oct 30 00:14:36.597839 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:36.597741 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-deda90ae-7b4f645874-jjjnn" event={"ID":"70171153-613b-4185-bfaa-1dd277298e44","Type":"ContainerStarted","Data":"25504cfce8d969c37872643d5bac753c936416438dccfc39d29d97ba89054948"} Oct 30 00:14:36.598020 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:36.597841 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-deda90ae-7b4f645874-jjjnn" Oct 30 00:14:36.642574 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:36.642521 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-deda90ae-7b4f645874-jjjnn" podStartSLOduration=1.52924026 podStartE2EDuration="3.642503308s" totalImagesPullingTime="2.113263048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:14:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:14:36.640337644 +0000 UTC m=+591.845838850" watchObservedRunningTime="2026-10-30 00:14:36.642503308 +0000 UTC m=+591.848004490" Oct 30 00:14:36.705827 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:36.705794 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-185595da-6f9d6ddd86-6cfxr"] Oct 30 00:14:36.709388 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:14:36.709358 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode76cf1c1_30bf_4823_a7c5_53380f123448.slice/crio-7c3868c3fc5b83badaef7b92e3063587122afabdd3e7755681559025887a0a27 WatchSource:0}: Error finding container 7c3868c3fc5b83badaef7b92e3063587122afabdd3e7755681559025887a0a27: Status 404 returned error can't find the container with id 7c3868c3fc5b83badaef7b92e3063587122afabdd3e7755681559025887a0a27 Oct 30 00:14:37.605861 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:37.605826 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-185595da-6f9d6ddd86-6cfxr" event={"ID":"e76cf1c1-30bf-4823-a7c5-53380f123448","Type":"ContainerStarted","Data":"74b834185e32b10d6c4b5b5b52b7792bebe0b438b280ce5e511ce938b207401b"} Oct 30 00:14:37.606272 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:37.605881 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-185595da-6f9d6ddd86-6cfxr" event={"ID":"e76cf1c1-30bf-4823-a7c5-53380f123448","Type":"ContainerStarted","Data":"7c3868c3fc5b83badaef7b92e3063587122afabdd3e7755681559025887a0a27"} Oct 30 00:14:37.606272 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:37.606050 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-185595da-6f9d6ddd86-6cfxr" Oct 30 00:14:37.642841 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:37.642792 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-185595da-6f9d6ddd86-6cfxr" podStartSLOduration=1.642778412 podStartE2EDuration="1.642778412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:14:37.640237602 +0000 UTC m=+592.845738784" watchObservedRunningTime="2026-10-30 00:14:37.642778412 +0000 UTC m=+592.848279592" Oct 30 00:14:38.476289 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:38.476252 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-6c45bb-546469c9db-sn7gc"] Oct 30 00:14:38.479876 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:38.479858 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-6c45bb-546469c9db-sn7gc" Oct 30 00:14:38.494558 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:38.494527 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-6c45bb\"" Oct 30 00:14:38.497872 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:38.497846 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-6c45bb-546469c9db-sn7gc"] Oct 30 00:14:38.576941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:38.576890 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-6c45bb-546469c9db-sn7gc\" (UID: \"e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-6c45bb-546469c9db-sn7gc" Oct 30 00:14:38.577183 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:38.576999 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9bnf\" (UniqueName: \"kubernetes.io/projected/e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e-kube-api-access-w9bnf\") pod \"payload-processing-e2e-worker-w6-6c45bb-546469c9db-sn7gc\" (UID: \"e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-6c45bb-546469c9db-sn7gc" Oct 30 00:14:38.678221 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:38.678184 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-6c45bb-546469c9db-sn7gc\" (UID: \"e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-6c45bb-546469c9db-sn7gc" Oct 30 00:14:38.678731 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:38.678251 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9bnf\" (UniqueName: \"kubernetes.io/projected/e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e-kube-api-access-w9bnf\") pod \"payload-processing-e2e-worker-w6-6c45bb-546469c9db-sn7gc\" (UID: \"e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-6c45bb-546469c9db-sn7gc" Oct 30 00:14:38.679195 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:38.679169 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-6c45bb-546469c9db-sn7gc\" (UID: \"e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-6c45bb-546469c9db-sn7gc" Oct 30 00:14:38.698845 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:38.698815 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9bnf\" (UniqueName: \"kubernetes.io/projected/e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e-kube-api-access-w9bnf\") pod \"payload-processing-e2e-worker-w6-6c45bb-546469c9db-sn7gc\" (UID: \"e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-6c45bb-546469c9db-sn7gc" Oct 30 00:14:38.793970 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:38.793939 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-6c45bb-546469c9db-sn7gc" Oct 30 00:14:39.091528 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:39.091455 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-6c45bb-546469c9db-sn7gc"] Oct 30 00:14:39.094991 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:14:39.094958 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode81b3fe0_9090_4d0b_94cc_6d2ce82fde4e.slice/crio-25ce406282dd1241f8889d9a3d226453ef043e2bf8a43d1b1a27ff3d2103de90 WatchSource:0}: Error finding container 25ce406282dd1241f8889d9a3d226453ef043e2bf8a43d1b1a27ff3d2103de90: Status 404 returned error can't find the container with id 25ce406282dd1241f8889d9a3d226453ef043e2bf8a43d1b1a27ff3d2103de90 Oct 30 00:14:39.539267 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:39.539226 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8ffe924e-5fd58cdcf8-pxzcm"] Oct 30 00:14:39.544015 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:39.543985 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8ffe924e-5fd58cdcf8-pxzcm" Oct 30 00:14:39.568854 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:39.568818 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8ffe924e-5fd58cdcf8-pxzcm"] Oct 30 00:14:39.574787 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:39.574758 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-8ffe924e\"" Oct 30 00:14:39.575013 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:39.574756 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-8ffe924e-dockercfg-hnzps\"" Oct 30 00:14:39.593245 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:39.593215 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpf6s\" (UniqueName: \"kubernetes.io/projected/336500a6-7517-4793-9e4c-64ea0e676fc6-kube-api-access-vpf6s\") pod \"payload-processing-e2e-migrate-8ffe924e-5fd58cdcf8-pxzcm\" (UID: \"336500a6-7517-4793-9e4c-64ea0e676fc6\") " pod="openshift-ingress/payload-processing-e2e-migrate-8ffe924e-5fd58cdcf8-pxzcm" Oct 30 00:14:39.593387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:39.593256 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/336500a6-7517-4793-9e4c-64ea0e676fc6-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8ffe924e-5fd58cdcf8-pxzcm\" (UID: \"336500a6-7517-4793-9e4c-64ea0e676fc6\") " pod="openshift-ingress/payload-processing-e2e-migrate-8ffe924e-5fd58cdcf8-pxzcm" Oct 30 00:14:39.633704 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:39.633672 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-6c45bb-546469c9db-sn7gc" event={"ID":"e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e","Type":"ContainerStarted","Data":"14a7d2d61ae1c3af08156d67e12b3f5280552cc84503079efd3111c2dc380b0f"} Oct 30 00:14:39.633864 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:39.633725 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-6c45bb-546469c9db-sn7gc" event={"ID":"e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e","Type":"ContainerStarted","Data":"25ce406282dd1241f8889d9a3d226453ef043e2bf8a43d1b1a27ff3d2103de90"} Oct 30 00:14:39.633864 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:39.633748 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-6c45bb-546469c9db-sn7gc" Oct 30 00:14:39.684110 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:39.671262 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-6c45bb-546469c9db-sn7gc" podStartSLOduration=1.6712414199999999 podStartE2EDuration="1.67124142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:14:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:14:39.669276877 +0000 UTC m=+594.874778059" watchObservedRunningTime="2026-10-30 00:14:39.67124142 +0000 UTC m=+594.876742598" Oct 30 00:14:39.695464 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:39.694740 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpf6s\" (UniqueName: \"kubernetes.io/projected/336500a6-7517-4793-9e4c-64ea0e676fc6-kube-api-access-vpf6s\") pod \"payload-processing-e2e-migrate-8ffe924e-5fd58cdcf8-pxzcm\" (UID: \"336500a6-7517-4793-9e4c-64ea0e676fc6\") " pod="openshift-ingress/payload-processing-e2e-migrate-8ffe924e-5fd58cdcf8-pxzcm" Oct 30 00:14:39.695464 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:39.694792 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/336500a6-7517-4793-9e4c-64ea0e676fc6-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8ffe924e-5fd58cdcf8-pxzcm\" (UID: \"336500a6-7517-4793-9e4c-64ea0e676fc6\") " pod="openshift-ingress/payload-processing-e2e-migrate-8ffe924e-5fd58cdcf8-pxzcm" Oct 30 00:14:39.695716 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:39.695663 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/336500a6-7517-4793-9e4c-64ea0e676fc6-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8ffe924e-5fd58cdcf8-pxzcm\" (UID: \"336500a6-7517-4793-9e4c-64ea0e676fc6\") " pod="openshift-ingress/payload-processing-e2e-migrate-8ffe924e-5fd58cdcf8-pxzcm" Oct 30 00:14:39.716747 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:39.716701 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpf6s\" (UniqueName: \"kubernetes.io/projected/336500a6-7517-4793-9e4c-64ea0e676fc6-kube-api-access-vpf6s\") pod \"payload-processing-e2e-migrate-8ffe924e-5fd58cdcf8-pxzcm\" (UID: \"336500a6-7517-4793-9e4c-64ea0e676fc6\") " pod="openshift-ingress/payload-processing-e2e-migrate-8ffe924e-5fd58cdcf8-pxzcm" Oct 30 00:14:39.868624 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:39.868539 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8ffe924e-5fd58cdcf8-pxzcm" Oct 30 00:14:40.249009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:40.244079 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8ffe924e-5fd58cdcf8-pxzcm"] Oct 30 00:14:40.259385 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:14:40.259343 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336500a6_7517_4793_9e4c_64ea0e676fc6.slice/crio-6f45ebe821867809562c40929ab14544585317c8b951256ad0dcc791fad577c8 WatchSource:0}: Error finding container 6f45ebe821867809562c40929ab14544585317c8b951256ad0dcc791fad577c8: Status 404 returned error can't find the container with id 6f45ebe821867809562c40929ab14544585317c8b951256ad0dcc791fad577c8 Oct 30 00:14:40.641186 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:40.641146 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8ffe924e-5fd58cdcf8-pxzcm" event={"ID":"336500a6-7517-4793-9e4c-64ea0e676fc6","Type":"ContainerStarted","Data":"82144cf03c0833c90eb119cb64398efc7795de7c2e033cdd8848b1bd6ad2efb6"} Oct 30 00:14:40.641340 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:40.641204 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8ffe924e-5fd58cdcf8-pxzcm" event={"ID":"336500a6-7517-4793-9e4c-64ea0e676fc6","Type":"ContainerStarted","Data":"6f45ebe821867809562c40929ab14544585317c8b951256ad0dcc791fad577c8"} Oct 30 00:14:40.641425 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:40.641363 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-8ffe924e-5fd58cdcf8-pxzcm" Oct 30 00:14:40.678352 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:40.678303 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-8ffe924e-5fd58cdcf8-pxzcm" podStartSLOduration=1.678287878 podStartE2EDuration="1.678287878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:14:40.676055754 +0000 UTC m=+595.881556935" watchObservedRunningTime="2026-10-30 00:14:40.678287878 +0000 UTC m=+595.883789058" Oct 30 00:14:41.213867 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:41.213835 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7505bd-76fc47c6d7-jqkf2"] Oct 30 00:14:41.217838 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:41.217817 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7505bd-76fc47c6d7-jqkf2" Oct 30 00:14:41.234404 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:41.234379 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-7505bd-dockercfg-g5x88\"" Oct 30 00:14:41.237014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:41.236989 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-7505bd\"" Oct 30 00:14:41.237165 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:41.237148 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7505bd-76fc47c6d7-jqkf2"] Oct 30 00:14:41.284265 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:41.284228 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7505bd-65fc9799fb-8p95v"] Oct 30 00:14:41.288532 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:41.288507 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7505bd-65fc9799fb-8p95v" Oct 30 00:14:41.309602 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:41.306849 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7505bd-65fc9799fb-8p95v"] Oct 30 00:14:41.318644 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:41.314124 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9306cc6a-b126-4e05-b726-fc9b267a0d78-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-7505bd-76fc47c6d7-jqkf2\" (UID: \"9306cc6a-b126-4e05-b726-fc9b267a0d78\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7505bd-76fc47c6d7-jqkf2" Oct 30 00:14:41.318644 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:41.314168 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcflx\" (UniqueName: \"kubernetes.io/projected/91e5f27d-f718-4239-98ee-b0a17382677e-kube-api-access-zcflx\") pod \"payload-processing-e2e-worker-w0-7505bd-65fc9799fb-8p95v\" (UID: \"91e5f27d-f718-4239-98ee-b0a17382677e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7505bd-65fc9799fb-8p95v" Oct 30 00:14:41.318644 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:41.314205 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwt87\" (UniqueName: \"kubernetes.io/projected/9306cc6a-b126-4e05-b726-fc9b267a0d78-kube-api-access-wwt87\") pod \"payload-pre-processing-e2e-worker-w0-7505bd-76fc47c6d7-jqkf2\" (UID: \"9306cc6a-b126-4e05-b726-fc9b267a0d78\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7505bd-76fc47c6d7-jqkf2" Oct 30 00:14:41.318644 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:41.314229 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e5f27d-f718-4239-98ee-b0a17382677e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-7505bd-65fc9799fb-8p95v\" (UID: \"91e5f27d-f718-4239-98ee-b0a17382677e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7505bd-65fc9799fb-8p95v" Oct 30 00:14:41.415572 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:41.415535 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9306cc6a-b126-4e05-b726-fc9b267a0d78-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-7505bd-76fc47c6d7-jqkf2\" (UID: \"9306cc6a-b126-4e05-b726-fc9b267a0d78\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7505bd-76fc47c6d7-jqkf2" Oct 30 00:14:41.415572 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:41.415574 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcflx\" (UniqueName: \"kubernetes.io/projected/91e5f27d-f718-4239-98ee-b0a17382677e-kube-api-access-zcflx\") pod \"payload-processing-e2e-worker-w0-7505bd-65fc9799fb-8p95v\" (UID: \"91e5f27d-f718-4239-98ee-b0a17382677e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7505bd-65fc9799fb-8p95v" Oct 30 00:14:41.415796 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:41.415594 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwt87\" (UniqueName: \"kubernetes.io/projected/9306cc6a-b126-4e05-b726-fc9b267a0d78-kube-api-access-wwt87\") pod \"payload-pre-processing-e2e-worker-w0-7505bd-76fc47c6d7-jqkf2\" (UID: \"9306cc6a-b126-4e05-b726-fc9b267a0d78\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7505bd-76fc47c6d7-jqkf2" Oct 30 00:14:41.415796 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:41.415620 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e5f27d-f718-4239-98ee-b0a17382677e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-7505bd-65fc9799fb-8p95v\" (UID: \"91e5f27d-f718-4239-98ee-b0a17382677e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7505bd-65fc9799fb-8p95v" Oct 30 00:14:41.416327 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:41.416308 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e5f27d-f718-4239-98ee-b0a17382677e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-7505bd-65fc9799fb-8p95v\" (UID: \"91e5f27d-f718-4239-98ee-b0a17382677e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7505bd-65fc9799fb-8p95v" Oct 30 00:14:41.416386 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:41.416353 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9306cc6a-b126-4e05-b726-fc9b267a0d78-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-7505bd-76fc47c6d7-jqkf2\" (UID: \"9306cc6a-b126-4e05-b726-fc9b267a0d78\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7505bd-76fc47c6d7-jqkf2" Oct 30 00:14:41.436419 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:41.436390 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcflx\" (UniqueName: \"kubernetes.io/projected/91e5f27d-f718-4239-98ee-b0a17382677e-kube-api-access-zcflx\") pod \"payload-processing-e2e-worker-w0-7505bd-65fc9799fb-8p95v\" (UID: \"91e5f27d-f718-4239-98ee-b0a17382677e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7505bd-65fc9799fb-8p95v" Oct 30 00:14:41.436533 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:41.436397 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwt87\" (UniqueName: \"kubernetes.io/projected/9306cc6a-b126-4e05-b726-fc9b267a0d78-kube-api-access-wwt87\") pod \"payload-pre-processing-e2e-worker-w0-7505bd-76fc47c6d7-jqkf2\" (UID: \"9306cc6a-b126-4e05-b726-fc9b267a0d78\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7505bd-76fc47c6d7-jqkf2" Oct 30 00:14:41.532038 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:41.532007 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7505bd-76fc47c6d7-jqkf2" Oct 30 00:14:41.611436 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:41.611411 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7505bd-65fc9799fb-8p95v" Oct 30 00:14:41.831026 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:41.830946 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7505bd-76fc47c6d7-jqkf2"] Oct 30 00:14:41.834067 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:14:41.834040 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9306cc6a_b126_4e05_b726_fc9b267a0d78.slice/crio-67dd33b011cf03f179004a7ea247e0ecb4dd45c5078e8414334b2778c870ee88 WatchSource:0}: Error finding container 67dd33b011cf03f179004a7ea247e0ecb4dd45c5078e8414334b2778c870ee88: Status 404 returned error can't find the container with id 67dd33b011cf03f179004a7ea247e0ecb4dd45c5078e8414334b2778c870ee88 Oct 30 00:14:41.939549 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:41.939520 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7505bd-65fc9799fb-8p95v"] Oct 30 00:14:41.942847 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:14:41.942821 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e5f27d_f718_4239_98ee_b0a17382677e.slice/crio-9115cfe89cf18da0dfb3305cdd5ea8f95fe819658aa3bd8116847599d093be1f WatchSource:0}: Error finding container 9115cfe89cf18da0dfb3305cdd5ea8f95fe819658aa3bd8116847599d093be1f: Status 404 returned error can't find the container with id 9115cfe89cf18da0dfb3305cdd5ea8f95fe819658aa3bd8116847599d093be1f Oct 30 00:14:42.654756 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:42.654719 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7505bd-65fc9799fb-8p95v" event={"ID":"91e5f27d-f718-4239-98ee-b0a17382677e","Type":"ContainerStarted","Data":"5100a7e1faf6eb7eecdc37f3085290ba0e4f6ac1174b4d266e516fd656972caa"} Oct 30 00:14:42.655237 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:42.654905 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-7505bd-65fc9799fb-8p95v" Oct 30 00:14:42.655237 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:42.654975 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7505bd-65fc9799fb-8p95v" event={"ID":"91e5f27d-f718-4239-98ee-b0a17382677e","Type":"ContainerStarted","Data":"9115cfe89cf18da0dfb3305cdd5ea8f95fe819658aa3bd8116847599d093be1f"} Oct 30 00:14:42.656538 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:42.656510 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7505bd-76fc47c6d7-jqkf2" event={"ID":"9306cc6a-b126-4e05-b726-fc9b267a0d78","Type":"ContainerStarted","Data":"3b50ee621aec95eeb34d5f21d11f52f1106a629b91f09552ea4bccbc8b6bff93"} Oct 30 00:14:42.656645 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:42.656559 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7505bd-76fc47c6d7-jqkf2" event={"ID":"9306cc6a-b126-4e05-b726-fc9b267a0d78","Type":"ContainerStarted","Data":"67dd33b011cf03f179004a7ea247e0ecb4dd45c5078e8414334b2778c870ee88"} Oct 30 00:14:42.656645 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:42.656602 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7505bd-76fc47c6d7-jqkf2" Oct 30 00:14:42.690110 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:42.690066 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-7505bd-65fc9799fb-8p95v" podStartSLOduration=1.690054274 podStartE2EDuration="1.690054274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:14:42.689043645 +0000 UTC m=+597.894544808" watchObservedRunningTime="2026-10-30 00:14:42.690054274 +0000 UTC m=+597.895555454" Oct 30 00:14:42.725474 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:42.725420 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7505bd-76fc47c6d7-jqkf2" podStartSLOduration=1.7254095390000002 podStartE2EDuration="1.725409539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:14:42.723536911 +0000 UTC m=+597.929038089" watchObservedRunningTime="2026-10-30 00:14:42.725409539 +0000 UTC m=+597.930910724" Oct 30 00:14:45.076458 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:45.076431 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8gb5_9fa688d8-29ca-4c1f-926a-f1a4099e6b44/console-operator/2.log" Oct 30 00:14:45.079227 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:45.079193 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8gb5_9fa688d8-29ca-4c1f-926a-f1a4099e6b44/console-operator/2.log" Oct 30 00:14:46.170213 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:46.170165 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-7a7ac6-68f8c58595-vb58b"] Oct 30 00:14:46.174968 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:46.174952 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a7ac6-68f8c58595-vb58b" Oct 30 00:14:46.198733 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:46.198703 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-7a7ac6\"" Oct 30 00:14:46.198866 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:46.198784 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-7a7ac6-dockercfg-flt7z\"" Oct 30 00:14:46.201588 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:46.201563 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-7a7ac6-68f8c58595-vb58b"] Oct 30 00:14:46.260532 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:46.260488 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5smbl\" (UniqueName: \"kubernetes.io/projected/776a2d19-ec0d-4868-ac0c-e22a7c4b6de0-kube-api-access-5smbl\") pod \"payload-pre-processing-e2e-worker-w1-7a7ac6-68f8c58595-vb58b\" (UID: \"776a2d19-ec0d-4868-ac0c-e22a7c4b6de0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a7ac6-68f8c58595-vb58b" Oct 30 00:14:46.260881 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:46.260858 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/776a2d19-ec0d-4868-ac0c-e22a7c4b6de0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-7a7ac6-68f8c58595-vb58b\" (UID: \"776a2d19-ec0d-4868-ac0c-e22a7c4b6de0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a7ac6-68f8c58595-vb58b" Oct 30 00:14:46.266172 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:46.266112 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7a7ac6-75b958fbc4-dqgk7"] Oct 30 00:14:46.270595 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:46.270565 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a7ac6-75b958fbc4-dqgk7" Oct 30 00:14:46.292164 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:46.292134 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7a7ac6-75b958fbc4-dqgk7"] Oct 30 00:14:46.362032 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:46.361988 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/776a2d19-ec0d-4868-ac0c-e22a7c4b6de0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-7a7ac6-68f8c58595-vb58b\" (UID: \"776a2d19-ec0d-4868-ac0c-e22a7c4b6de0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a7ac6-68f8c58595-vb58b" Oct 30 00:14:46.362238 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:46.362103 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5smbl\" (UniqueName: \"kubernetes.io/projected/776a2d19-ec0d-4868-ac0c-e22a7c4b6de0-kube-api-access-5smbl\") pod \"payload-pre-processing-e2e-worker-w1-7a7ac6-68f8c58595-vb58b\" (UID: \"776a2d19-ec0d-4868-ac0c-e22a7c4b6de0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a7ac6-68f8c58595-vb58b" Oct 30 00:14:46.363891 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:46.363860 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/776a2d19-ec0d-4868-ac0c-e22a7c4b6de0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-7a7ac6-68f8c58595-vb58b\" (UID: \"776a2d19-ec0d-4868-ac0c-e22a7c4b6de0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a7ac6-68f8c58595-vb58b" Oct 30 00:14:46.383113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:46.383088 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5smbl\" (UniqueName: \"kubernetes.io/projected/776a2d19-ec0d-4868-ac0c-e22a7c4b6de0-kube-api-access-5smbl\") pod \"payload-pre-processing-e2e-worker-w1-7a7ac6-68f8c58595-vb58b\" (UID: \"776a2d19-ec0d-4868-ac0c-e22a7c4b6de0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a7ac6-68f8c58595-vb58b" Oct 30 00:14:46.463303 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:46.463215 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lcqn\" (UniqueName: \"kubernetes.io/projected/30bf357c-11ff-4fc6-bcfe-c776a3370b9f-kube-api-access-8lcqn\") pod \"payload-processing-e2e-worker-w1-7a7ac6-75b958fbc4-dqgk7\" (UID: \"30bf357c-11ff-4fc6-bcfe-c776a3370b9f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7a7ac6-75b958fbc4-dqgk7" Oct 30 00:14:46.463303 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:46.463260 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30bf357c-11ff-4fc6-bcfe-c776a3370b9f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-7a7ac6-75b958fbc4-dqgk7\" (UID: \"30bf357c-11ff-4fc6-bcfe-c776a3370b9f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7a7ac6-75b958fbc4-dqgk7" Oct 30 00:14:46.487858 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:46.487833 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a7ac6-68f8c58595-vb58b" Oct 30 00:14:46.564554 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:46.564521 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lcqn\" (UniqueName: \"kubernetes.io/projected/30bf357c-11ff-4fc6-bcfe-c776a3370b9f-kube-api-access-8lcqn\") pod \"payload-processing-e2e-worker-w1-7a7ac6-75b958fbc4-dqgk7\" (UID: \"30bf357c-11ff-4fc6-bcfe-c776a3370b9f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7a7ac6-75b958fbc4-dqgk7" Oct 30 00:14:46.564734 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:46.564565 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30bf357c-11ff-4fc6-bcfe-c776a3370b9f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-7a7ac6-75b958fbc4-dqgk7\" (UID: \"30bf357c-11ff-4fc6-bcfe-c776a3370b9f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7a7ac6-75b958fbc4-dqgk7" Oct 30 00:14:46.565479 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:46.565456 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30bf357c-11ff-4fc6-bcfe-c776a3370b9f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-7a7ac6-75b958fbc4-dqgk7\" (UID: \"30bf357c-11ff-4fc6-bcfe-c776a3370b9f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7a7ac6-75b958fbc4-dqgk7" Oct 30 00:14:46.585186 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:46.585154 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lcqn\" (UniqueName: \"kubernetes.io/projected/30bf357c-11ff-4fc6-bcfe-c776a3370b9f-kube-api-access-8lcqn\") pod \"payload-processing-e2e-worker-w1-7a7ac6-75b958fbc4-dqgk7\" (UID: \"30bf357c-11ff-4fc6-bcfe-c776a3370b9f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7a7ac6-75b958fbc4-dqgk7" Oct 30 00:14:46.603083 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:46.603055 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a7ac6-75b958fbc4-dqgk7" Oct 30 00:14:46.781486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:46.781450 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-7a7ac6-68f8c58595-vb58b"] Oct 30 00:14:46.784715 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:14:46.784688 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776a2d19_ec0d_4868_ac0c_e22a7c4b6de0.slice/crio-7a5b4d0dc7623448759816b9eacb0dd080eb06d4ce946a9dad9b7a1c01279a0e WatchSource:0}: Error finding container 7a5b4d0dc7623448759816b9eacb0dd080eb06d4ce946a9dad9b7a1c01279a0e: Status 404 returned error can't find the container with id 7a5b4d0dc7623448759816b9eacb0dd080eb06d4ce946a9dad9b7a1c01279a0e Oct 30 00:14:46.938020 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:46.937989 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7a7ac6-75b958fbc4-dqgk7"] Oct 30 00:14:46.941140 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:14:46.941114 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30bf357c_11ff_4fc6_bcfe_c776a3370b9f.slice/crio-396f25f16fcb6997bb3cce4c8ff9716bf448ea71f6c4fdd763643096b75e928e WatchSource:0}: Error finding container 396f25f16fcb6997bb3cce4c8ff9716bf448ea71f6c4fdd763643096b75e928e: Status 404 returned error can't find the container with id 396f25f16fcb6997bb3cce4c8ff9716bf448ea71f6c4fdd763643096b75e928e Oct 30 00:14:47.322355 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:14:47.322319 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:14:47.609407 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:47.609331 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-deda90ae-7b4f645874-jjjnn" Oct 30 00:14:47.692595 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:47.692557 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a7ac6-68f8c58595-vb58b" event={"ID":"776a2d19-ec0d-4868-ac0c-e22a7c4b6de0","Type":"ContainerStarted","Data":"14c8a467de7b1d26a1a5858f03e940d85a951858ef045595a7184ff2c77a57d8"} Oct 30 00:14:47.692801 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:47.692621 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a7ac6-68f8c58595-vb58b" event={"ID":"776a2d19-ec0d-4868-ac0c-e22a7c4b6de0","Type":"ContainerStarted","Data":"7a5b4d0dc7623448759816b9eacb0dd080eb06d4ce946a9dad9b7a1c01279a0e"} Oct 30 00:14:47.692801 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:47.692747 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a7ac6-68f8c58595-vb58b" Oct 30 00:14:47.694246 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:47.694212 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a7ac6-75b958fbc4-dqgk7" event={"ID":"30bf357c-11ff-4fc6-bcfe-c776a3370b9f","Type":"ContainerStarted","Data":"f2583b68cc5f20b95025592720f04fabb73f3a6d76b0377ba09a08df4a935a5d"} Oct 30 00:14:47.694373 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:47.694268 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a7ac6-75b958fbc4-dqgk7" Oct 30 00:14:47.694373 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:47.694292 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a7ac6-75b958fbc4-dqgk7" event={"ID":"30bf357c-11ff-4fc6-bcfe-c776a3370b9f","Type":"ContainerStarted","Data":"396f25f16fcb6997bb3cce4c8ff9716bf448ea71f6c4fdd763643096b75e928e"} Oct 30 00:14:47.727727 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:47.727679 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a7ac6-68f8c58595-vb58b" podStartSLOduration=1.7276658710000001 podStartE2EDuration="1.727665871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:14:47.72737456 +0000 UTC m=+602.932875742" watchObservedRunningTime="2026-10-30 00:14:47.727665871 +0000 UTC m=+602.933167052" Oct 30 00:14:47.764324 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:47.764281 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a7ac6-75b958fbc4-dqgk7" podStartSLOduration=1.7642687129999999 podStartE2EDuration="1.764268713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:14:47.763298198 +0000 UTC m=+602.968799372" watchObservedRunningTime="2026-10-30 00:14:47.764268713 +0000 UTC m=+602.969769953" Oct 30 00:14:48.630950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:48.630907 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-185595da-6f9d6ddd86-6cfxr" Oct 30 00:14:50.645187 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:50.645157 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-6c45bb-546469c9db-sn7gc" Oct 30 00:14:51.650070 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:51.650042 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-8ffe924e-5fd58cdcf8-pxzcm" Oct 30 00:14:51.714509 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:51.714477 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-deda90ae-7kffx"] Oct 30 00:14:51.718341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:51.718325 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-deda90ae-7kffx" Oct 30 00:14:51.724287 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:51.724261 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aae647ba-86db-430e-9c5a-ed0b8cb22707-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-deda90ae-7kffx\" (UID: \"aae647ba-86db-430e-9c5a-ed0b8cb22707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-deda90ae-7kffx" Oct 30 00:14:51.724419 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:51.724305 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aae647ba-86db-430e-9c5a-ed0b8cb22707-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-deda90ae-7kffx\" (UID: \"aae647ba-86db-430e-9c5a-ed0b8cb22707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-deda90ae-7kffx" Oct 30 00:14:51.724500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:51.724484 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwphv\" (UniqueName: \"kubernetes.io/projected/aae647ba-86db-430e-9c5a-ed0b8cb22707-kube-api-access-zwphv\") pod \"maas-api-revoke-keys-e2e-derive-deda90ae-7kffx\" (UID: \"aae647ba-86db-430e-9c5a-ed0b8cb22707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-deda90ae-7kffx" Oct 30 00:14:51.730847 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:51.730821 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-deda90ae-7kffx"] Oct 30 00:14:51.732484 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:51.732464 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:14:51.737119 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:51.737099 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:14:51.737396 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:51.737377 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:14:51.737493 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:51.737437 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:14:51.825458 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:51.825426 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aae647ba-86db-430e-9c5a-ed0b8cb22707-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-deda90ae-7kffx\" (UID: \"aae647ba-86db-430e-9c5a-ed0b8cb22707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-deda90ae-7kffx" Oct 30 00:14:51.825599 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:51.825472 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aae647ba-86db-430e-9c5a-ed0b8cb22707-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-deda90ae-7kffx\" (UID: \"aae647ba-86db-430e-9c5a-ed0b8cb22707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-deda90ae-7kffx" Oct 30 00:14:51.825599 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:51.825544 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwphv\" (UniqueName: \"kubernetes.io/projected/aae647ba-86db-430e-9c5a-ed0b8cb22707-kube-api-access-zwphv\") pod \"maas-api-revoke-keys-e2e-derive-deda90ae-7kffx\" (UID: \"aae647ba-86db-430e-9c5a-ed0b8cb22707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-deda90ae-7kffx" Oct 30 00:14:51.826247 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:51.826217 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aae647ba-86db-430e-9c5a-ed0b8cb22707-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-deda90ae-7kffx\" (UID: \"aae647ba-86db-430e-9c5a-ed0b8cb22707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-deda90ae-7kffx" Oct 30 00:14:51.827840 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:51.827823 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aae647ba-86db-430e-9c5a-ed0b8cb22707-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-deda90ae-7kffx\" (UID: \"aae647ba-86db-430e-9c5a-ed0b8cb22707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-deda90ae-7kffx" Oct 30 00:14:51.846081 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:51.846056 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwphv\" (UniqueName: \"kubernetes.io/projected/aae647ba-86db-430e-9c5a-ed0b8cb22707-kube-api-access-zwphv\") pod \"maas-api-revoke-keys-e2e-derive-deda90ae-7kffx\" (UID: \"aae647ba-86db-430e-9c5a-ed0b8cb22707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-deda90ae-7kffx" Oct 30 00:14:52.034756 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:52.034718 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-deda90ae-7kffx" Oct 30 00:14:52.329496 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:52.329428 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-deda90ae-7kffx"] Oct 30 00:14:52.333308 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:14:52.333284 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaae647ba_86db_430e_9c5a_ed0b8cb22707.slice/crio-5af65ddd7aa953626b3c954bd5e98308b21bc8984677e2421f52769a883e91eb WatchSource:0}: Error finding container 5af65ddd7aa953626b3c954bd5e98308b21bc8984677e2421f52769a883e91eb: Status 404 returned error can't find the container with id 5af65ddd7aa953626b3c954bd5e98308b21bc8984677e2421f52769a883e91eb Oct 30 00:14:52.745313 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:52.745232 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-deda90ae-7kffx" event={"ID":"aae647ba-86db-430e-9c5a-ed0b8cb22707","Type":"ContainerStarted","Data":"5af65ddd7aa953626b3c954bd5e98308b21bc8984677e2421f52769a883e91eb"} Oct 30 00:14:53.047719 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:53.047690 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-185595da-xq2pp"] Oct 30 00:14:53.052793 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:53.052775 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-185595da-xq2pp" Oct 30 00:14:53.069354 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:53.069326 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-185595da-xq2pp"] Oct 30 00:14:53.134877 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:53.134842 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmxp4\" (UniqueName: \"kubernetes.io/projected/31e047e4-bc04-4d68-945e-39251841a051-kube-api-access-qmxp4\") pod \"maas-api-revoke-keys-e2e-ait-185595da-xq2pp\" (UID: \"31e047e4-bc04-4d68-945e-39251841a051\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-185595da-xq2pp" Oct 30 00:14:53.134877 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:53.134885 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31e047e4-bc04-4d68-945e-39251841a051-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-185595da-xq2pp\" (UID: \"31e047e4-bc04-4d68-945e-39251841a051\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-185595da-xq2pp" Oct 30 00:14:53.135121 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:53.134928 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31e047e4-bc04-4d68-945e-39251841a051-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-185595da-xq2pp\" (UID: \"31e047e4-bc04-4d68-945e-39251841a051\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-185595da-xq2pp" Oct 30 00:14:53.235948 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:53.235901 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmxp4\" (UniqueName: \"kubernetes.io/projected/31e047e4-bc04-4d68-945e-39251841a051-kube-api-access-qmxp4\") pod \"maas-api-revoke-keys-e2e-ait-185595da-xq2pp\" (UID: \"31e047e4-bc04-4d68-945e-39251841a051\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-185595da-xq2pp" Oct 30 00:14:53.236136 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:53.235956 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31e047e4-bc04-4d68-945e-39251841a051-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-185595da-xq2pp\" (UID: \"31e047e4-bc04-4d68-945e-39251841a051\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-185595da-xq2pp" Oct 30 00:14:53.236136 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:53.235995 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31e047e4-bc04-4d68-945e-39251841a051-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-185595da-xq2pp\" (UID: \"31e047e4-bc04-4d68-945e-39251841a051\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-185595da-xq2pp" Oct 30 00:14:53.236846 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:53.236815 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31e047e4-bc04-4d68-945e-39251841a051-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-185595da-xq2pp\" (UID: \"31e047e4-bc04-4d68-945e-39251841a051\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-185595da-xq2pp" Oct 30 00:14:53.238533 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:53.238509 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31e047e4-bc04-4d68-945e-39251841a051-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-185595da-xq2pp\" (UID: \"31e047e4-bc04-4d68-945e-39251841a051\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-185595da-xq2pp" Oct 30 00:14:53.256715 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:53.256686 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmxp4\" (UniqueName: \"kubernetes.io/projected/31e047e4-bc04-4d68-945e-39251841a051-kube-api-access-qmxp4\") pod \"maas-api-revoke-keys-e2e-ait-185595da-xq2pp\" (UID: \"31e047e4-bc04-4d68-945e-39251841a051\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-185595da-xq2pp" Oct 30 00:14:53.366077 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:53.365993 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-185595da-xq2pp" Oct 30 00:14:53.665240 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:53.665154 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-7505bd-65fc9799fb-8p95v" Oct 30 00:14:53.665240 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:53.665227 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7505bd-76fc47c6d7-jqkf2" Oct 30 00:14:53.673713 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:53.673681 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-185595da-xq2pp"] Oct 30 00:14:53.677038 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:14:53.677008 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31e047e4_bc04_4d68_945e_39251841a051.slice/crio-3feebde4def9ef3ea44eaf30c38f1b4c39b8fc700c97bf86d5a9a198112972d7 WatchSource:0}: Error finding container 3feebde4def9ef3ea44eaf30c38f1b4c39b8fc700c97bf86d5a9a198112972d7: Status 404 returned error can't find the container with id 3feebde4def9ef3ea44eaf30c38f1b4c39b8fc700c97bf86d5a9a198112972d7 Oct 30 00:14:53.755224 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:53.755183 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-185595da-xq2pp" event={"ID":"31e047e4-bc04-4d68-945e-39251841a051","Type":"ContainerStarted","Data":"3feebde4def9ef3ea44eaf30c38f1b4c39b8fc700c97bf86d5a9a198112972d7"} Oct 30 00:14:55.769045 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:55.769008 5001 generic.go:356] "Generic (PLEG): container finished" podID="aae647ba-86db-430e-9c5a-ed0b8cb22707" containerID="5f55214eb6a394e36c0e0361b03d0af1641c77c3efafd82cbff5d94050765e6a" exitCode=0 Oct 30 00:14:55.769445 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:55.769099 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-deda90ae-7kffx" event={"ID":"aae647ba-86db-430e-9c5a-ed0b8cb22707","Type":"ContainerDied","Data":"5f55214eb6a394e36c0e0361b03d0af1641c77c3efafd82cbff5d94050765e6a"} Oct 30 00:14:55.770965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:55.770938 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-185595da-xq2pp" event={"ID":"31e047e4-bc04-4d68-945e-39251841a051","Type":"ContainerStarted","Data":"24814a0b004d54d3cd15cfbf2dae0a6eaee087c5c1ee33b2dc6e2a1b6ae53ef7"} Oct 30 00:14:55.839116 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:55.839064 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-185595da-xq2pp" podStartSLOduration=1.399691506 podStartE2EDuration="2.839051885s" totalImagesPullingTime="1.439360379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:14:55.837942954 +0000 UTC m=+611.043444136" watchObservedRunningTime="2026-10-30 00:14:55.839051885 +0000 UTC m=+611.044553067" Oct 30 00:14:56.778173 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:56.778137 5001 generic.go:356] "Generic (PLEG): container finished" podID="31e047e4-bc04-4d68-945e-39251841a051" containerID="24814a0b004d54d3cd15cfbf2dae0a6eaee087c5c1ee33b2dc6e2a1b6ae53ef7" exitCode=0 Oct 30 00:14:56.778663 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:56.778224 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-185595da-xq2pp" event={"ID":"31e047e4-bc04-4d68-945e-39251841a051","Type":"ContainerDied","Data":"24814a0b004d54d3cd15cfbf2dae0a6eaee087c5c1ee33b2dc6e2a1b6ae53ef7"} Oct 30 00:14:56.940499 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:56.940472 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-deda90ae-7kffx" Oct 30 00:14:56.975880 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:56.975853 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aae647ba-86db-430e-9c5a-ed0b8cb22707-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aae647ba-86db-430e-9c5a-ed0b8cb22707-maas-api-openshift-ca\") pod \"aae647ba-86db-430e-9c5a-ed0b8cb22707\" (UID: \"aae647ba-86db-430e-9c5a-ed0b8cb22707\") " Oct 30 00:14:56.976025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:56.975901 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aae647ba-86db-430e-9c5a-ed0b8cb22707-kube-api-access-zwphv\" (UniqueName: \"kubernetes.io/projected/aae647ba-86db-430e-9c5a-ed0b8cb22707-kube-api-access-zwphv\") pod \"aae647ba-86db-430e-9c5a-ed0b8cb22707\" (UID: \"aae647ba-86db-430e-9c5a-ed0b8cb22707\") " Oct 30 00:14:56.976271 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:56.976244 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aae647ba-86db-430e-9c5a-ed0b8cb22707-maas-api-openshift-ca" pod "aae647ba-86db-430e-9c5a-ed0b8cb22707" (UID: "aae647ba-86db-430e-9c5a-ed0b8cb22707"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:14:56.978118 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:56.978097 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aae647ba-86db-430e-9c5a-ed0b8cb22707-kube-api-access-zwphv" pod "aae647ba-86db-430e-9c5a-ed0b8cb22707" (UID: "aae647ba-86db-430e-9c5a-ed0b8cb22707"). InnerVolumeSpecName "kube-api-access-zwphv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:14:57.076453 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.076370 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aae647ba-86db-430e-9c5a-ed0b8cb22707-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aae647ba-86db-430e-9c5a-ed0b8cb22707-maas-api-xks-ca\") pod \"aae647ba-86db-430e-9c5a-ed0b8cb22707\" (UID: \"aae647ba-86db-430e-9c5a-ed0b8cb22707\") " Oct 30 00:14:57.076712 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.076695 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aae647ba-86db-430e-9c5a-ed0b8cb22707-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:14:57.076793 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.076718 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwphv\" (UniqueName: \"kubernetes.io/projected/aae647ba-86db-430e-9c5a-ed0b8cb22707-kube-api-access-zwphv\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:14:57.078566 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.078533 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aae647ba-86db-430e-9c5a-ed0b8cb22707-maas-api-xks-ca" pod "aae647ba-86db-430e-9c5a-ed0b8cb22707" (UID: "aae647ba-86db-430e-9c5a-ed0b8cb22707"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:14:57.177216 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.177174 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aae647ba-86db-430e-9c5a-ed0b8cb22707-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:14:57.574175 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.574140 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv"] Oct 30 00:14:57.574953 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.574932 5001 cpu_manager.go:414] "containerMap: removing container" podUID="aae647ba-86db-430e-9c5a-ed0b8cb22707" containerName="revoke-keys" Oct 30 00:14:57.575050 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.574956 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aae647ba-86db-430e-9c5a-ed0b8cb22707" containerName="revoke-keys" Oct 30 00:14:57.575109 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.575096 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="aae647ba-86db-430e-9c5a-ed0b8cb22707" containerName="revoke-keys" Oct 30 00:14:57.578538 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.578517 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:14:57.580366 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.580330 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef5c38fa-99ee-44ee-a734-793b583df3e4-home\") pod \"simulator-w0-7505bd-kserve-6496f76dc6-gdzsv\" (UID: \"ef5c38fa-99ee-44ee-a734-793b583df3e4\") " pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:14:57.580487 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.580457 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef5c38fa-99ee-44ee-a734-793b583df3e4-dshm\") pod \"simulator-w0-7505bd-kserve-6496f76dc6-gdzsv\" (UID: \"ef5c38fa-99ee-44ee-a734-793b583df3e4\") " pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:14:57.580568 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.580509 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef5c38fa-99ee-44ee-a734-793b583df3e4-model-cache\") pod \"simulator-w0-7505bd-kserve-6496f76dc6-gdzsv\" (UID: \"ef5c38fa-99ee-44ee-a734-793b583df3e4\") " pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:14:57.580568 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.580544 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef5c38fa-99ee-44ee-a734-793b583df3e4-tmp-dir\") pod \"simulator-w0-7505bd-kserve-6496f76dc6-gdzsv\" (UID: \"ef5c38fa-99ee-44ee-a734-793b583df3e4\") " pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:14:57.580674 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.580575 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvr9d\" (UniqueName: \"kubernetes.io/projected/ef5c38fa-99ee-44ee-a734-793b583df3e4-kube-api-access-bvr9d\") pod \"simulator-w0-7505bd-kserve-6496f76dc6-gdzsv\" (UID: \"ef5c38fa-99ee-44ee-a734-793b583df3e4\") " pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:14:57.580731 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.580721 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef5c38fa-99ee-44ee-a734-793b583df3e4-tls-certs\") pod \"simulator-w0-7505bd-kserve-6496f76dc6-gdzsv\" (UID: \"ef5c38fa-99ee-44ee-a734-793b583df3e4\") " pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:14:57.607433 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.607401 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7505bd\"/\"openshift-service-ca.crt\"" Oct 30 00:14:57.607685 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.607669 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7505bd\"/\"default-dockercfg-s9svd\"" Oct 30 00:14:57.607856 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.607842 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7505bd\"/\"kube-root-ca.crt\"" Oct 30 00:14:57.608076 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.608050 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7505bd\"/\"simulator-w0-7505bd-kserve-self-signed-certs\"" Oct 30 00:14:57.611153 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.611130 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv"] Oct 30 00:14:57.681697 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.681657 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef5c38fa-99ee-44ee-a734-793b583df3e4-tls-certs\") pod \"simulator-w0-7505bd-kserve-6496f76dc6-gdzsv\" (UID: \"ef5c38fa-99ee-44ee-a734-793b583df3e4\") " pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:14:57.681852 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.681717 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef5c38fa-99ee-44ee-a734-793b583df3e4-home\") pod \"simulator-w0-7505bd-kserve-6496f76dc6-gdzsv\" (UID: \"ef5c38fa-99ee-44ee-a734-793b583df3e4\") " pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:14:57.681907 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.681890 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef5c38fa-99ee-44ee-a734-793b583df3e4-dshm\") pod \"simulator-w0-7505bd-kserve-6496f76dc6-gdzsv\" (UID: \"ef5c38fa-99ee-44ee-a734-793b583df3e4\") " pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:14:57.681971 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.681956 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef5c38fa-99ee-44ee-a734-793b583df3e4-model-cache\") pod \"simulator-w0-7505bd-kserve-6496f76dc6-gdzsv\" (UID: \"ef5c38fa-99ee-44ee-a734-793b583df3e4\") " pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:14:57.682282 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.681992 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef5c38fa-99ee-44ee-a734-793b583df3e4-tmp-dir\") pod \"simulator-w0-7505bd-kserve-6496f76dc6-gdzsv\" (UID: \"ef5c38fa-99ee-44ee-a734-793b583df3e4\") " pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:14:57.682282 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.682017 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvr9d\" (UniqueName: \"kubernetes.io/projected/ef5c38fa-99ee-44ee-a734-793b583df3e4-kube-api-access-bvr9d\") pod \"simulator-w0-7505bd-kserve-6496f76dc6-gdzsv\" (UID: \"ef5c38fa-99ee-44ee-a734-793b583df3e4\") " pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:14:57.682282 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.682156 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef5c38fa-99ee-44ee-a734-793b583df3e4-home\") pod \"simulator-w0-7505bd-kserve-6496f76dc6-gdzsv\" (UID: \"ef5c38fa-99ee-44ee-a734-793b583df3e4\") " pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:14:57.682552 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.682502 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef5c38fa-99ee-44ee-a734-793b583df3e4-tmp-dir\") pod \"simulator-w0-7505bd-kserve-6496f76dc6-gdzsv\" (UID: \"ef5c38fa-99ee-44ee-a734-793b583df3e4\") " pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:14:57.682892 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.682861 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef5c38fa-99ee-44ee-a734-793b583df3e4-model-cache\") pod \"simulator-w0-7505bd-kserve-6496f76dc6-gdzsv\" (UID: \"ef5c38fa-99ee-44ee-a734-793b583df3e4\") " pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:14:57.684240 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.684217 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef5c38fa-99ee-44ee-a734-793b583df3e4-dshm\") pod \"simulator-w0-7505bd-kserve-6496f76dc6-gdzsv\" (UID: \"ef5c38fa-99ee-44ee-a734-793b583df3e4\") " pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:14:57.684900 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.684876 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef5c38fa-99ee-44ee-a734-793b583df3e4-tls-certs\") pod \"simulator-w0-7505bd-kserve-6496f76dc6-gdzsv\" (UID: \"ef5c38fa-99ee-44ee-a734-793b583df3e4\") " pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:14:57.702813 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.702788 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvr9d\" (UniqueName: \"kubernetes.io/projected/ef5c38fa-99ee-44ee-a734-793b583df3e4-kube-api-access-bvr9d\") pod \"simulator-w0-7505bd-kserve-6496f76dc6-gdzsv\" (UID: \"ef5c38fa-99ee-44ee-a734-793b583df3e4\") " pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:14:57.784941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.784883 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-deda90ae-7kffx" event={"ID":"aae647ba-86db-430e-9c5a-ed0b8cb22707","Type":"ContainerDied","Data":"5af65ddd7aa953626b3c954bd5e98308b21bc8984677e2421f52769a883e91eb"} Oct 30 00:14:57.784941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.784948 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5af65ddd7aa953626b3c954bd5e98308b21bc8984677e2421f52769a883e91eb" Oct 30 00:14:57.785385 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.784899 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-deda90ae-7kffx" Oct 30 00:14:57.835999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.831975 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c9e9fa7-9lrln"] Oct 30 00:14:57.850104 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.850074 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c9e9fa7-9lrln"] Oct 30 00:14:57.850422 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.850406 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c9e9fa7-9lrln" Oct 30 00:14:57.883624 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.883590 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtn76\" (UniqueName: \"kubernetes.io/projected/8b311c78-77ca-485b-a3e5-5dd368bbb94b-kube-api-access-gtn76\") pod \"maas-api-revoke-keys-e2e-ait-6c9e9fa7-9lrln\" (UID: \"8b311c78-77ca-485b-a3e5-5dd368bbb94b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c9e9fa7-9lrln" Oct 30 00:14:57.883864 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.883838 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b311c78-77ca-485b-a3e5-5dd368bbb94b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6c9e9fa7-9lrln\" (UID: \"8b311c78-77ca-485b-a3e5-5dd368bbb94b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c9e9fa7-9lrln" Oct 30 00:14:57.884001 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.883897 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b311c78-77ca-485b-a3e5-5dd368bbb94b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6c9e9fa7-9lrln\" (UID: \"8b311c78-77ca-485b-a3e5-5dd368bbb94b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c9e9fa7-9lrln" Oct 30 00:14:57.894212 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.894191 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:14:57.961173 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.961145 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-185595da-xq2pp" Oct 30 00:14:57.985193 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.985165 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31e047e4-bc04-4d68-945e-39251841a051-kube-api-access-qmxp4\" (UniqueName: \"kubernetes.io/projected/31e047e4-bc04-4d68-945e-39251841a051-kube-api-access-qmxp4\") pod \"31e047e4-bc04-4d68-945e-39251841a051\" (UID: \"31e047e4-bc04-4d68-945e-39251841a051\") " Oct 30 00:14:57.985364 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.985263 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31e047e4-bc04-4d68-945e-39251841a051-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31e047e4-bc04-4d68-945e-39251841a051-maas-api-openshift-ca\") pod \"31e047e4-bc04-4d68-945e-39251841a051\" (UID: \"31e047e4-bc04-4d68-945e-39251841a051\") " Oct 30 00:14:57.985364 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.985311 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31e047e4-bc04-4d68-945e-39251841a051-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31e047e4-bc04-4d68-945e-39251841a051-maas-api-xks-ca\") pod \"31e047e4-bc04-4d68-945e-39251841a051\" (UID: \"31e047e4-bc04-4d68-945e-39251841a051\") " Oct 30 00:14:57.985620 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.985589 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31e047e4-bc04-4d68-945e-39251841a051-maas-api-openshift-ca" pod "31e047e4-bc04-4d68-945e-39251841a051" (UID: "31e047e4-bc04-4d68-945e-39251841a051"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:14:57.985620 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.985619 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtn76\" (UniqueName: \"kubernetes.io/projected/8b311c78-77ca-485b-a3e5-5dd368bbb94b-kube-api-access-gtn76\") pod \"maas-api-revoke-keys-e2e-ait-6c9e9fa7-9lrln\" (UID: \"8b311c78-77ca-485b-a3e5-5dd368bbb94b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c9e9fa7-9lrln" Oct 30 00:14:57.985791 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.985679 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b311c78-77ca-485b-a3e5-5dd368bbb94b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6c9e9fa7-9lrln\" (UID: \"8b311c78-77ca-485b-a3e5-5dd368bbb94b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c9e9fa7-9lrln" Oct 30 00:14:57.985791 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.985730 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b311c78-77ca-485b-a3e5-5dd368bbb94b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6c9e9fa7-9lrln\" (UID: \"8b311c78-77ca-485b-a3e5-5dd368bbb94b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c9e9fa7-9lrln" Oct 30 00:14:57.985791 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.985779 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31e047e4-bc04-4d68-945e-39251841a051-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:14:57.986613 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.986588 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b311c78-77ca-485b-a3e5-5dd368bbb94b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6c9e9fa7-9lrln\" (UID: \"8b311c78-77ca-485b-a3e5-5dd368bbb94b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c9e9fa7-9lrln" Oct 30 00:14:57.987404 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.987382 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31e047e4-bc04-4d68-945e-39251841a051-kube-api-access-qmxp4" pod "31e047e4-bc04-4d68-945e-39251841a051" (UID: "31e047e4-bc04-4d68-945e-39251841a051"). InnerVolumeSpecName "kube-api-access-qmxp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:14:57.987709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.987695 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31e047e4-bc04-4d68-945e-39251841a051-maas-api-xks-ca" pod "31e047e4-bc04-4d68-945e-39251841a051" (UID: "31e047e4-bc04-4d68-945e-39251841a051"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:14:57.988153 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:57.988135 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b311c78-77ca-485b-a3e5-5dd368bbb94b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6c9e9fa7-9lrln\" (UID: \"8b311c78-77ca-485b-a3e5-5dd368bbb94b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c9e9fa7-9lrln" Oct 30 00:14:58.009180 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:58.009151 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtn76\" (UniqueName: \"kubernetes.io/projected/8b311c78-77ca-485b-a3e5-5dd368bbb94b-kube-api-access-gtn76\") pod \"maas-api-revoke-keys-e2e-ait-6c9e9fa7-9lrln\" (UID: \"8b311c78-77ca-485b-a3e5-5dd368bbb94b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c9e9fa7-9lrln" Oct 30 00:14:58.086712 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:58.086635 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmxp4\" (UniqueName: \"kubernetes.io/projected/31e047e4-bc04-4d68-945e-39251841a051-kube-api-access-qmxp4\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:14:58.086712 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:58.086662 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31e047e4-bc04-4d68-945e-39251841a051-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:14:58.165186 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:58.165157 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c9e9fa7-9lrln" Oct 30 00:14:58.200694 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:58.200660 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv"] Oct 30 00:14:58.207285 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:14:58.207248 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef5c38fa_99ee_44ee_a734_793b583df3e4.slice/crio-9cb5625134841c69d5396762b364a297af2b41ab74a89be9c0aafb93a886ffaa WatchSource:0}: Error finding container 9cb5625134841c69d5396762b364a297af2b41ab74a89be9c0aafb93a886ffaa: Status 404 returned error can't find the container with id 9cb5625134841c69d5396762b364a297af2b41ab74a89be9c0aafb93a886ffaa Oct 30 00:14:58.209438 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:58.209420 5001 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:14:58.477077 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:58.476984 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c9e9fa7-9lrln"] Oct 30 00:14:58.486018 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:14:58.485983 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b311c78_77ca_485b_a3e5_5dd368bbb94b.slice/crio-a1abb8a355e4a255a8b2e9bbd8ddd4e1b0a74e047444ea3ddccba7b8f1508447 WatchSource:0}: Error finding container a1abb8a355e4a255a8b2e9bbd8ddd4e1b0a74e047444ea3ddccba7b8f1508447: Status 404 returned error can't find the container with id a1abb8a355e4a255a8b2e9bbd8ddd4e1b0a74e047444ea3ddccba7b8f1508447 Oct 30 00:14:58.705793 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:58.705760 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a7ac6-75b958fbc4-dqgk7" Oct 30 00:14:58.706356 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:58.706335 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a7ac6-68f8c58595-vb58b" Oct 30 00:14:58.795084 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:58.795044 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" event={"ID":"ef5c38fa-99ee-44ee-a734-793b583df3e4","Type":"ContainerStarted","Data":"9cb5625134841c69d5396762b364a297af2b41ab74a89be9c0aafb93a886ffaa"} Oct 30 00:14:58.796855 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:58.796826 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-185595da-xq2pp" Oct 30 00:14:58.796855 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:58.796843 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-185595da-xq2pp" event={"ID":"31e047e4-bc04-4d68-945e-39251841a051","Type":"ContainerDied","Data":"3feebde4def9ef3ea44eaf30c38f1b4c39b8fc700c97bf86d5a9a198112972d7"} Oct 30 00:14:58.797078 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:58.796887 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3feebde4def9ef3ea44eaf30c38f1b4c39b8fc700c97bf86d5a9a198112972d7" Oct 30 00:14:58.806936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:58.806867 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c9e9fa7-9lrln" event={"ID":"8b311c78-77ca-485b-a3e5-5dd368bbb94b","Type":"ContainerStarted","Data":"a1abb8a355e4a255a8b2e9bbd8ddd4e1b0a74e047444ea3ddccba7b8f1508447"} Oct 30 00:14:59.814967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:59.814860 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" event={"ID":"ef5c38fa-99ee-44ee-a734-793b583df3e4","Type":"ContainerStarted","Data":"04f40afe893364fdcd34cb0a4a45118deff02fefee6e0795a66323fcfc996803"} Oct 30 00:14:59.816508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:59.816481 5001 generic.go:356] "Generic (PLEG): container finished" podID="8b311c78-77ca-485b-a3e5-5dd368bbb94b" containerID="2c11bc80ded7787fbf161f3658b1dae8a1ccf78549c587486e82a847fc52b9c6" exitCode=0 Oct 30 00:14:59.816604 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:59.816560 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c9e9fa7-9lrln" event={"ID":"8b311c78-77ca-485b-a3e5-5dd368bbb94b","Type":"ContainerDied","Data":"2c11bc80ded7787fbf161f3658b1dae8a1ccf78549c587486e82a847fc52b9c6"} Oct 30 00:14:59.852024 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:59.851978 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podStartSLOduration=1.535688908 podStartE2EDuration="2.851963978s" totalImagesPullingTime="1.31627507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:14:59.850178871 +0000 UTC m=+615.055680051" watchObservedRunningTime="2026-10-30 00:14:59.851963978 +0000 UTC m=+615.057465191" Oct 30 00:14:59.894312 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:14:59.894284 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:15:00.151293 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.151195 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6"] Oct 30 00:15:00.151902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.151872 5001 cpu_manager.go:414] "containerMap: removing container" podUID="31e047e4-bc04-4d68-945e-39251841a051" containerName="revoke-keys" Oct 30 00:15:00.151902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.151897 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31e047e4-bc04-4d68-945e-39251841a051" containerName="revoke-keys" Oct 30 00:15:00.152133 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.152078 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="31e047e4-bc04-4d68-945e-39251841a051" containerName="revoke-keys" Oct 30 00:15:00.164206 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.164175 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-deda90ae-29888655-k45hj"] Oct 30 00:15:00.164333 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.164285 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6" Oct 30 00:15:00.168290 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.168259 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-185595da-29888655-crpzq"] Oct 30 00:15:00.168436 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.168334 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-deda90ae-29888655-k45hj" Oct 30 00:15:00.172270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.172248 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-185595da-29888655-crpzq" Oct 30 00:15:00.178586 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.178561 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6"] Oct 30 00:15:00.185575 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.183278 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89"] Oct 30 00:15:00.188207 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.188186 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89" Oct 30 00:15:00.189552 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.189525 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-deda90ae-29888655-k45hj"] Oct 30 00:15:00.193529 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.193505 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-185595da-29888655-crpzq"] Oct 30 00:15:00.218907 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.218874 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89"] Oct 30 00:15:00.219738 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.219713 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75ca90e5-c780-4851-92e0-f3619116c5f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89\" (UID: \"75ca90e5-c780-4851-92e0-f3619116c5f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89" Oct 30 00:15:00.219844 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.219806 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6\" (UID: \"7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6" Oct 30 00:15:00.219897 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.219849 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvzt5\" (UniqueName: \"kubernetes.io/projected/75ca90e5-c780-4851-92e0-f3619116c5f3-kube-api-access-vvzt5\") pod \"maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89\" (UID: \"75ca90e5-c780-4851-92e0-f3619116c5f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89" Oct 30 00:15:00.219980 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.219958 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdk88\" (UniqueName: \"kubernetes.io/projected/fdf0d73b-bbdc-4399-b32b-b5d2b887374d-kube-api-access-zdk88\") pod \"maas-api-key-cleanup-e2e-derive-deda90ae-29888655-k45hj\" (UID: \"fdf0d73b-bbdc-4399-b32b-b5d2b887374d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-deda90ae-29888655-k45hj" Oct 30 00:15:00.220040 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.219995 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf0d73b-bbdc-4399-b32b-b5d2b887374d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-deda90ae-29888655-k45hj\" (UID: \"fdf0d73b-bbdc-4399-b32b-b5d2b887374d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-deda90ae-29888655-k45hj" Oct 30 00:15:00.220040 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.220021 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e608f82-1172-468b-9537-7f1841f0488e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-185595da-29888655-crpzq\" (UID: \"1e608f82-1172-468b-9537-7f1841f0488e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-185595da-29888655-crpzq" Oct 30 00:15:00.220188 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.220172 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9jwd\" (UniqueName: \"kubernetes.io/projected/7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d-kube-api-access-x9jwd\") pod \"maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6\" (UID: \"7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6" Oct 30 00:15:00.220275 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.220228 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr6z2\" (UniqueName: \"kubernetes.io/projected/1e608f82-1172-468b-9537-7f1841f0488e-kube-api-access-sr6z2\") pod \"maas-api-key-cleanup-e2e-ait-185595da-29888655-crpzq\" (UID: \"1e608f82-1172-468b-9537-7f1841f0488e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-185595da-29888655-crpzq" Oct 30 00:15:00.321043 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.321008 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvzt5\" (UniqueName: \"kubernetes.io/projected/75ca90e5-c780-4851-92e0-f3619116c5f3-kube-api-access-vvzt5\") pod \"maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89\" (UID: \"75ca90e5-c780-4851-92e0-f3619116c5f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89" Oct 30 00:15:00.321043 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.321051 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdk88\" (UniqueName: \"kubernetes.io/projected/fdf0d73b-bbdc-4399-b32b-b5d2b887374d-kube-api-access-zdk88\") pod \"maas-api-key-cleanup-e2e-derive-deda90ae-29888655-k45hj\" (UID: \"fdf0d73b-bbdc-4399-b32b-b5d2b887374d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-deda90ae-29888655-k45hj" Oct 30 00:15:00.321254 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.321070 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf0d73b-bbdc-4399-b32b-b5d2b887374d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-deda90ae-29888655-k45hj\" (UID: \"fdf0d73b-bbdc-4399-b32b-b5d2b887374d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-deda90ae-29888655-k45hj" Oct 30 00:15:00.321254 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.321100 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e608f82-1172-468b-9537-7f1841f0488e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-185595da-29888655-crpzq\" (UID: \"1e608f82-1172-468b-9537-7f1841f0488e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-185595da-29888655-crpzq" Oct 30 00:15:00.321254 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.321167 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9jwd\" (UniqueName: \"kubernetes.io/projected/7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d-kube-api-access-x9jwd\") pod \"maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6\" (UID: \"7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6" Oct 30 00:15:00.321254 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.321217 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr6z2\" (UniqueName: \"kubernetes.io/projected/1e608f82-1172-468b-9537-7f1841f0488e-kube-api-access-sr6z2\") pod \"maas-api-key-cleanup-e2e-ait-185595da-29888655-crpzq\" (UID: \"1e608f82-1172-468b-9537-7f1841f0488e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-185595da-29888655-crpzq" Oct 30 00:15:00.321463 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.321262 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75ca90e5-c780-4851-92e0-f3619116c5f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89\" (UID: \"75ca90e5-c780-4851-92e0-f3619116c5f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89" Oct 30 00:15:00.321463 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.321306 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6\" (UID: \"7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6" Oct 30 00:15:00.322035 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.322005 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e608f82-1172-468b-9537-7f1841f0488e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-185595da-29888655-crpzq\" (UID: \"1e608f82-1172-468b-9537-7f1841f0488e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-185595da-29888655-crpzq" Oct 30 00:15:00.322035 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.322021 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf0d73b-bbdc-4399-b32b-b5d2b887374d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-deda90ae-29888655-k45hj\" (UID: \"fdf0d73b-bbdc-4399-b32b-b5d2b887374d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-deda90ae-29888655-k45hj" Oct 30 00:15:00.322195 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.322021 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6\" (UID: \"7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6" Oct 30 00:15:00.322195 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.322156 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75ca90e5-c780-4851-92e0-f3619116c5f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89\" (UID: \"75ca90e5-c780-4851-92e0-f3619116c5f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89" Oct 30 00:15:00.341738 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.341714 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvzt5\" (UniqueName: \"kubernetes.io/projected/75ca90e5-c780-4851-92e0-f3619116c5f3-kube-api-access-vvzt5\") pod \"maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89\" (UID: \"75ca90e5-c780-4851-92e0-f3619116c5f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89" Oct 30 00:15:00.341863 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.341840 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdk88\" (UniqueName: \"kubernetes.io/projected/fdf0d73b-bbdc-4399-b32b-b5d2b887374d-kube-api-access-zdk88\") pod \"maas-api-key-cleanup-e2e-derive-deda90ae-29888655-k45hj\" (UID: \"fdf0d73b-bbdc-4399-b32b-b5d2b887374d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-deda90ae-29888655-k45hj" Oct 30 00:15:00.341950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.341936 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9jwd\" (UniqueName: \"kubernetes.io/projected/7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d-kube-api-access-x9jwd\") pod \"maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6\" (UID: \"7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6" Oct 30 00:15:00.341995 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.341968 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr6z2\" (UniqueName: \"kubernetes.io/projected/1e608f82-1172-468b-9537-7f1841f0488e-kube-api-access-sr6z2\") pod \"maas-api-key-cleanup-e2e-ait-185595da-29888655-crpzq\" (UID: \"1e608f82-1172-468b-9537-7f1841f0488e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-185595da-29888655-crpzq" Oct 30 00:15:00.486825 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.486746 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6" Oct 30 00:15:00.495977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.495959 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-deda90ae-29888655-k45hj" Oct 30 00:15:00.504329 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.504306 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-185595da-29888655-crpzq" Oct 30 00:15:00.522998 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.522966 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89" Oct 30 00:15:00.796530 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.796500 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6"] Oct 30 00:15:00.801505 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:15:00.801481 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb42483_aef9_4c39_a4f7_0e6a9b7c6e8d.slice/crio-0219ed2d81aa3cf7f85a85f8a3f4b4fcbdad41810d26ed7c27c3f63b393f2601 WatchSource:0}: Error finding container 0219ed2d81aa3cf7f85a85f8a3f4b4fcbdad41810d26ed7c27c3f63b393f2601: Status 404 returned error can't find the container with id 0219ed2d81aa3cf7f85a85f8a3f4b4fcbdad41810d26ed7c27c3f63b393f2601 Oct 30 00:15:00.808971 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.808947 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-deda90ae-29888655-k45hj"] Oct 30 00:15:00.813855 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:15:00.813831 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf0d73b_bbdc_4399_b32b_b5d2b887374d.slice/crio-886d141831651b446550cb5778a84b0eb8d231004e44b3cfba7beb3f8dc0c249 WatchSource:0}: Error finding container 886d141831651b446550cb5778a84b0eb8d231004e44b3cfba7beb3f8dc0c249: Status 404 returned error can't find the container with id 886d141831651b446550cb5778a84b0eb8d231004e44b3cfba7beb3f8dc0c249 Oct 30 00:15:00.821536 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.821513 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-185595da-29888655-crpzq"] Oct 30 00:15:00.825179 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.825152 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-deda90ae-29888655-k45hj" event={"ID":"fdf0d73b-bbdc-4399-b32b-b5d2b887374d","Type":"ContainerStarted","Data":"886d141831651b446550cb5778a84b0eb8d231004e44b3cfba7beb3f8dc0c249"} Oct 30 00:15:00.826736 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.826698 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6" event={"ID":"7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d","Type":"ContainerStarted","Data":"0219ed2d81aa3cf7f85a85f8a3f4b4fcbdad41810d26ed7c27c3f63b393f2601"} Oct 30 00:15:00.827542 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:15:00.827497 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e608f82_1172_468b_9537_7f1841f0488e.slice/crio-f5b1f7fb582c3b6ddbe57ace1ea6f0597bea9a49efc82c824c55988b21d5c9db WatchSource:0}: Error finding container f5b1f7fb582c3b6ddbe57ace1ea6f0597bea9a49efc82c824c55988b21d5c9db: Status 404 returned error can't find the container with id f5b1f7fb582c3b6ddbe57ace1ea6f0597bea9a49efc82c824c55988b21d5c9db Oct 30 00:15:00.847289 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:00.847266 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89"] Oct 30 00:15:00.849650 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:15:00.849626 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75ca90e5_c780_4851_92e0_f3619116c5f3.slice/crio-681d7b514fefb9d51d7774196d01d02fa99b13b34212c73d5b46c16b58a2be9c WatchSource:0}: Error finding container 681d7b514fefb9d51d7774196d01d02fa99b13b34212c73d5b46c16b58a2be9c: Status 404 returned error can't find the container with id 681d7b514fefb9d51d7774196d01d02fa99b13b34212c73d5b46c16b58a2be9c Oct 30 00:15:01.034803 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:01.034779 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c9e9fa7-9lrln" Oct 30 00:15:01.129129 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:01.129058 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b311c78-77ca-485b-a3e5-5dd368bbb94b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b311c78-77ca-485b-a3e5-5dd368bbb94b-maas-api-xks-ca\") pod \"8b311c78-77ca-485b-a3e5-5dd368bbb94b\" (UID: \"8b311c78-77ca-485b-a3e5-5dd368bbb94b\") " Oct 30 00:15:01.129129 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:01.129093 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b311c78-77ca-485b-a3e5-5dd368bbb94b-kube-api-access-gtn76\" (UniqueName: \"kubernetes.io/projected/8b311c78-77ca-485b-a3e5-5dd368bbb94b-kube-api-access-gtn76\") pod \"8b311c78-77ca-485b-a3e5-5dd368bbb94b\" (UID: \"8b311c78-77ca-485b-a3e5-5dd368bbb94b\") " Oct 30 00:15:01.129129 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:01.129130 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b311c78-77ca-485b-a3e5-5dd368bbb94b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b311c78-77ca-485b-a3e5-5dd368bbb94b-maas-api-openshift-ca\") pod \"8b311c78-77ca-485b-a3e5-5dd368bbb94b\" (UID: \"8b311c78-77ca-485b-a3e5-5dd368bbb94b\") " Oct 30 00:15:01.129522 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:01.129495 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b311c78-77ca-485b-a3e5-5dd368bbb94b-maas-api-openshift-ca" pod "8b311c78-77ca-485b-a3e5-5dd368bbb94b" (UID: "8b311c78-77ca-485b-a3e5-5dd368bbb94b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:15:01.131094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:01.131067 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b311c78-77ca-485b-a3e5-5dd368bbb94b-maas-api-xks-ca" pod "8b311c78-77ca-485b-a3e5-5dd368bbb94b" (UID: "8b311c78-77ca-485b-a3e5-5dd368bbb94b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:15:01.131199 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:01.131182 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b311c78-77ca-485b-a3e5-5dd368bbb94b-kube-api-access-gtn76" pod "8b311c78-77ca-485b-a3e5-5dd368bbb94b" (UID: "8b311c78-77ca-485b-a3e5-5dd368bbb94b"). InnerVolumeSpecName "kube-api-access-gtn76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:15:01.229799 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:01.229771 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b311c78-77ca-485b-a3e5-5dd368bbb94b-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:15:01.229799 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:01.229800 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtn76\" (UniqueName: \"kubernetes.io/projected/8b311c78-77ca-485b-a3e5-5dd368bbb94b-kube-api-access-gtn76\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:15:01.229942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:01.229810 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b311c78-77ca-485b-a3e5-5dd368bbb94b-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:15:01.836012 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:01.835976 5001 generic.go:356] "Generic (PLEG): container finished" podID="75ca90e5-c780-4851-92e0-f3619116c5f3" containerID="afec88c997712c196ab3d9fa3032ec00e24493422e7144d4b4cad9052d6ec552" exitCode=60 Oct 30 00:15:01.836409 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:01.836057 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89" event={"ID":"75ca90e5-c780-4851-92e0-f3619116c5f3","Type":"ContainerDied","Data":"afec88c997712c196ab3d9fa3032ec00e24493422e7144d4b4cad9052d6ec552"} Oct 30 00:15:01.836409 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:01.836113 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89" event={"ID":"75ca90e5-c780-4851-92e0-f3619116c5f3","Type":"ContainerStarted","Data":"681d7b514fefb9d51d7774196d01d02fa99b13b34212c73d5b46c16b58a2be9c"} Oct 30 00:15:01.836409 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:01.836296 5001 scope.go:122] "RemoveContainer" containerID="afec88c997712c196ab3d9fa3032ec00e24493422e7144d4b4cad9052d6ec552" Oct 30 00:15:01.837842 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:01.837819 5001 generic.go:356] "Generic (PLEG): container finished" podID="7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d" containerID="033879d7961476119569cd796d1a10751c74aca63c5f75aed7cce5026c400cb7" exitCode=60 Oct 30 00:15:01.837946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:01.837842 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6" event={"ID":"7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d","Type":"ContainerDied","Data":"033879d7961476119569cd796d1a10751c74aca63c5f75aed7cce5026c400cb7"} Oct 30 00:15:01.838120 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:01.838097 5001 scope.go:122] "RemoveContainer" containerID="033879d7961476119569cd796d1a10751c74aca63c5f75aed7cce5026c400cb7" Oct 30 00:15:01.840113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:01.840091 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-deda90ae-29888655-k45hj" event={"ID":"fdf0d73b-bbdc-4399-b32b-b5d2b887374d","Type":"ContainerStarted","Data":"e38e4e7f6f4c4959e4908c9c9d030b0570af291973dd2446e55a30ce07c0363d"} Oct 30 00:15:01.841552 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:01.841530 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c9e9fa7-9lrln" event={"ID":"8b311c78-77ca-485b-a3e5-5dd368bbb94b","Type":"ContainerDied","Data":"a1abb8a355e4a255a8b2e9bbd8ddd4e1b0a74e047444ea3ddccba7b8f1508447"} Oct 30 00:15:01.841629 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:01.841573 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1abb8a355e4a255a8b2e9bbd8ddd4e1b0a74e047444ea3ddccba7b8f1508447" Oct 30 00:15:01.841629 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:01.841541 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c9e9fa7-9lrln" Oct 30 00:15:01.843044 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:01.843022 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-185595da-29888655-crpzq" event={"ID":"1e608f82-1172-468b-9537-7f1841f0488e","Type":"ContainerStarted","Data":"b9b36f9e7f72266306e3dc2c24a0d9069c899c2da49d2d8310f13ad8c4b3bb6a"} Oct 30 00:15:01.843145 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:01.843063 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-185595da-29888655-crpzq" event={"ID":"1e608f82-1172-468b-9537-7f1841f0488e","Type":"ContainerStarted","Data":"f5b1f7fb582c3b6ddbe57ace1ea6f0597bea9a49efc82c824c55988b21d5c9db"} Oct 30 00:15:01.906357 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:01.906306 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-185595da-29888655-crpzq" podStartSLOduration=1.906291475 podStartE2EDuration="1.906291475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:15:01.903780062 +0000 UTC m=+617.109281244" watchObservedRunningTime="2026-10-30 00:15:01.906291475 +0000 UTC m=+617.111792687" Oct 30 00:15:01.971866 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:01.971812 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-deda90ae-29888655-k45hj" podStartSLOduration=1.971799081 podStartE2EDuration="1.971799081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:15:01.970610344 +0000 UTC m=+617.176111525" watchObservedRunningTime="2026-10-30 00:15:01.971799081 +0000 UTC m=+617.177300264" Oct 30 00:15:02.042029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:02.042001 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-deda90ae-7b4f645874-jjjnn"] Oct 30 00:15:02.851683 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:02.851652 5001 generic.go:356] "Generic (PLEG): container finished" podID="fdf0d73b-bbdc-4399-b32b-b5d2b887374d" containerID="e38e4e7f6f4c4959e4908c9c9d030b0570af291973dd2446e55a30ce07c0363d" exitCode=60 Oct 30 00:15:02.852173 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:02.851750 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-deda90ae-29888655-k45hj" event={"ID":"fdf0d73b-bbdc-4399-b32b-b5d2b887374d","Type":"ContainerDied","Data":"e38e4e7f6f4c4959e4908c9c9d030b0570af291973dd2446e55a30ce07c0363d"} Oct 30 00:15:02.852173 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:02.852022 5001 scope.go:122] "RemoveContainer" containerID="e38e4e7f6f4c4959e4908c9c9d030b0570af291973dd2446e55a30ce07c0363d" Oct 30 00:15:02.853511 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:02.853485 5001 generic.go:356] "Generic (PLEG): container finished" podID="1e608f82-1172-468b-9537-7f1841f0488e" containerID="b9b36f9e7f72266306e3dc2c24a0d9069c899c2da49d2d8310f13ad8c4b3bb6a" exitCode=60 Oct 30 00:15:02.853595 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:02.853563 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-185595da-29888655-crpzq" event={"ID":"1e608f82-1172-468b-9537-7f1841f0488e","Type":"ContainerDied","Data":"b9b36f9e7f72266306e3dc2c24a0d9069c899c2da49d2d8310f13ad8c4b3bb6a"} Oct 30 00:15:02.853858 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:02.853844 5001 scope.go:122] "RemoveContainer" containerID="b9b36f9e7f72266306e3dc2c24a0d9069c899c2da49d2d8310f13ad8c4b3bb6a" Oct 30 00:15:02.856188 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:02.856168 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89" event={"ID":"75ca90e5-c780-4851-92e0-f3619116c5f3","Type":"ContainerStarted","Data":"246aca4957b0fe7e6931f3cee857c7992c11f6a1c57729ea9901e60248c8d1da"} Oct 30 00:15:02.859284 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:02.858187 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6" event={"ID":"7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d","Type":"ContainerStarted","Data":"12fb01a0d1d1cc4b8b59bbb0aff4de8b9fd1089be454039eb470667845cfae5b"} Oct 30 00:15:02.936769 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:02.936714 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89" podStartSLOduration=2.936695951 podStartE2EDuration="2.936695951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:15:02.93586645 +0000 UTC m=+618.141367633" watchObservedRunningTime="2026-10-30 00:15:02.936695951 +0000 UTC m=+618.142197133" Oct 30 00:15:03.007822 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:03.007772 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6" podStartSLOduration=3.007757924 podStartE2EDuration="3.007757924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:15:03.006603915 +0000 UTC m=+618.212105096" watchObservedRunningTime="2026-10-30 00:15:03.007757924 +0000 UTC m=+618.213259105" Oct 30 00:15:03.374146 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:03.374118 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ffe924e-z9vx8"] Oct 30 00:15:03.374605 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:03.374592 5001 cpu_manager.go:414] "containerMap: removing container" podUID="8b311c78-77ca-485b-a3e5-5dd368bbb94b" containerName="revoke-keys" Oct 30 00:15:03.374653 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:03.374606 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b311c78-77ca-485b-a3e5-5dd368bbb94b" containerName="revoke-keys" Oct 30 00:15:03.374686 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:03.374672 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b311c78-77ca-485b-a3e5-5dd368bbb94b" containerName="revoke-keys" Oct 30 00:15:03.377963 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:03.377942 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ffe924e-z9vx8" Oct 30 00:15:03.392623 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:03.392576 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:15:03.396035 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:03.396011 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ffe924e-z9vx8"] Oct 30 00:15:03.449682 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:03.449608 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b2c3136-aeca-4974-bc6c-0086f1f2fcda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8ffe924e-z9vx8\" (UID: \"7b2c3136-aeca-4974-bc6c-0086f1f2fcda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ffe924e-z9vx8" Oct 30 00:15:03.449682 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:03.449674 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz2dg\" (UniqueName: \"kubernetes.io/projected/7b2c3136-aeca-4974-bc6c-0086f1f2fcda-kube-api-access-vz2dg\") pod \"maas-api-revoke-keys-e2e-migrate-8ffe924e-z9vx8\" (UID: \"7b2c3136-aeca-4974-bc6c-0086f1f2fcda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ffe924e-z9vx8" Oct 30 00:15:03.449841 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:03.449781 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b2c3136-aeca-4974-bc6c-0086f1f2fcda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8ffe924e-z9vx8\" (UID: \"7b2c3136-aeca-4974-bc6c-0086f1f2fcda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ffe924e-z9vx8" Oct 30 00:15:03.550293 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:03.550256 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz2dg\" (UniqueName: \"kubernetes.io/projected/7b2c3136-aeca-4974-bc6c-0086f1f2fcda-kube-api-access-vz2dg\") pod \"maas-api-revoke-keys-e2e-migrate-8ffe924e-z9vx8\" (UID: \"7b2c3136-aeca-4974-bc6c-0086f1f2fcda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ffe924e-z9vx8" Oct 30 00:15:03.550478 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:03.550321 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b2c3136-aeca-4974-bc6c-0086f1f2fcda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8ffe924e-z9vx8\" (UID: \"7b2c3136-aeca-4974-bc6c-0086f1f2fcda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ffe924e-z9vx8" Oct 30 00:15:03.550478 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:03.550360 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b2c3136-aeca-4974-bc6c-0086f1f2fcda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8ffe924e-z9vx8\" (UID: \"7b2c3136-aeca-4974-bc6c-0086f1f2fcda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ffe924e-z9vx8" Oct 30 00:15:03.551139 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:03.551118 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b2c3136-aeca-4974-bc6c-0086f1f2fcda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8ffe924e-z9vx8\" (UID: \"7b2c3136-aeca-4974-bc6c-0086f1f2fcda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ffe924e-z9vx8" Oct 30 00:15:03.552968 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:03.552947 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b2c3136-aeca-4974-bc6c-0086f1f2fcda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8ffe924e-z9vx8\" (UID: \"7b2c3136-aeca-4974-bc6c-0086f1f2fcda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ffe924e-z9vx8" Oct 30 00:15:03.570764 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:03.570735 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz2dg\" (UniqueName: \"kubernetes.io/projected/7b2c3136-aeca-4974-bc6c-0086f1f2fcda-kube-api-access-vz2dg\") pod \"maas-api-revoke-keys-e2e-migrate-8ffe924e-z9vx8\" (UID: \"7b2c3136-aeca-4974-bc6c-0086f1f2fcda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ffe924e-z9vx8" Oct 30 00:15:03.690828 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:03.690799 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ffe924e-z9vx8" Oct 30 00:15:03.840347 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:03.840319 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-185595da-6f9d6ddd86-6cfxr"] Oct 30 00:15:03.866830 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:03.866795 5001 generic.go:356] "Generic (PLEG): container finished" podID="7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d" containerID="12fb01a0d1d1cc4b8b59bbb0aff4de8b9fd1089be454039eb470667845cfae5b" exitCode=60 Oct 30 00:15:03.867276 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:03.867094 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6" event={"ID":"7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d","Type":"ContainerDied","Data":"12fb01a0d1d1cc4b8b59bbb0aff4de8b9fd1089be454039eb470667845cfae5b"} Oct 30 00:15:03.867276 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:03.867162 5001 scope.go:122] "RemoveContainer" containerID="12fb01a0d1d1cc4b8b59bbb0aff4de8b9fd1089be454039eb470667845cfae5b" Oct 30 00:15:03.867387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:03.867304 5001 scope.go:122] "RemoveContainer" containerID="033879d7961476119569cd796d1a10751c74aca63c5f75aed7cce5026c400cb7" Oct 30 00:15:03.867457 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:15:03.867425 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6_odh-ai-gateway-infra(7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6" podUID="7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d" Oct 30 00:15:03.869861 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:03.869840 5001 generic.go:356] "Generic (PLEG): container finished" podID="fdf0d73b-bbdc-4399-b32b-b5d2b887374d" containerID="c51097cc18b8f057470234639978a2c5c5c75afeefed0504944763b5fcb593e5" exitCode=60 Oct 30 00:15:03.869981 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:03.869952 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-deda90ae-29888655-k45hj" event={"ID":"fdf0d73b-bbdc-4399-b32b-b5d2b887374d","Type":"ContainerDied","Data":"c51097cc18b8f057470234639978a2c5c5c75afeefed0504944763b5fcb593e5"} Oct 30 00:15:03.870176 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:03.870158 5001 scope.go:122] "RemoveContainer" containerID="c51097cc18b8f057470234639978a2c5c5c75afeefed0504944763b5fcb593e5" Oct 30 00:15:03.870407 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:15:03.870385 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-deda90ae-29888655-k45hj_odh-ai-gateway-infra(fdf0d73b-bbdc-4399-b32b-b5d2b887374d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-deda90ae-29888655-k45hj" podUID="fdf0d73b-bbdc-4399-b32b-b5d2b887374d" Oct 30 00:15:03.872168 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:03.872143 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-185595da-29888655-crpzq" event={"ID":"1e608f82-1172-468b-9537-7f1841f0488e","Type":"ContainerStarted","Data":"727ff9b27d8e9e7e1a84812f80faa9232cc56ad5f49201b3ed23aeec641703d5"} Oct 30 00:15:03.874759 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:03.874732 5001 generic.go:356] "Generic (PLEG): container finished" podID="75ca90e5-c780-4851-92e0-f3619116c5f3" containerID="246aca4957b0fe7e6931f3cee857c7992c11f6a1c57729ea9901e60248c8d1da" exitCode=60 Oct 30 00:15:03.874894 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:03.874770 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89" event={"ID":"75ca90e5-c780-4851-92e0-f3619116c5f3","Type":"ContainerDied","Data":"246aca4957b0fe7e6931f3cee857c7992c11f6a1c57729ea9901e60248c8d1da"} Oct 30 00:15:03.875105 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:03.875087 5001 scope.go:122] "RemoveContainer" containerID="246aca4957b0fe7e6931f3cee857c7992c11f6a1c57729ea9901e60248c8d1da" Oct 30 00:15:03.876418 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:15:03.875293 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89_odh-ai-gateway-infra(75ca90e5-c780-4851-92e0-f3619116c5f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89" podUID="75ca90e5-c780-4851-92e0-f3619116c5f3" Oct 30 00:15:03.883368 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:03.883354 5001 scope.go:122] "RemoveContainer" containerID="e38e4e7f6f4c4959e4908c9c9d030b0570af291973dd2446e55a30ce07c0363d" Oct 30 00:15:03.904078 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:03.904054 5001 scope.go:122] "RemoveContainer" containerID="afec88c997712c196ab3d9fa3032ec00e24493422e7144d4b4cad9052d6ec552" Oct 30 00:15:04.059786 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:04.059751 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ffe924e-z9vx8"] Oct 30 00:15:04.063043 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:15:04.063014 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b2c3136_aeca_4974_bc6c_0086f1f2fcda.slice/crio-c19f9d48570669ceaa3ec824a738c5ddbfd27e1fb13b0edd846433c4e04eb94b WatchSource:0}: Error finding container c19f9d48570669ceaa3ec824a738c5ddbfd27e1fb13b0edd846433c4e04eb94b: Status 404 returned error can't find the container with id c19f9d48570669ceaa3ec824a738c5ddbfd27e1fb13b0edd846433c4e04eb94b Oct 30 00:15:04.393439 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:04.393402 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-deda90ae-29888655-k45hj"] Oct 30 00:15:04.883871 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:04.883841 5001 scope.go:122] "RemoveContainer" containerID="246aca4957b0fe7e6931f3cee857c7992c11f6a1c57729ea9901e60248c8d1da" Oct 30 00:15:04.884407 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:15:04.884122 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89_odh-ai-gateway-infra(75ca90e5-c780-4851-92e0-f3619116c5f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89" podUID="75ca90e5-c780-4851-92e0-f3619116c5f3" Oct 30 00:15:04.885380 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:04.885358 5001 scope.go:122] "RemoveContainer" containerID="12fb01a0d1d1cc4b8b59bbb0aff4de8b9fd1089be454039eb470667845cfae5b" Oct 30 00:15:04.885637 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:15:04.885607 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6_odh-ai-gateway-infra(7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6" podUID="7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d" Oct 30 00:15:04.886874 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:04.886848 5001 scope.go:122] "RemoveContainer" containerID="c51097cc18b8f057470234639978a2c5c5c75afeefed0504944763b5fcb593e5" Oct 30 00:15:04.887093 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:15:04.887072 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-deda90ae-29888655-k45hj_odh-ai-gateway-infra(fdf0d73b-bbdc-4399-b32b-b5d2b887374d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-deda90ae-29888655-k45hj" podUID="fdf0d73b-bbdc-4399-b32b-b5d2b887374d" Oct 30 00:15:04.888619 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:04.888596 5001 generic.go:356] "Generic (PLEG): container finished" podID="1e608f82-1172-468b-9537-7f1841f0488e" containerID="727ff9b27d8e9e7e1a84812f80faa9232cc56ad5f49201b3ed23aeec641703d5" exitCode=60 Oct 30 00:15:04.888724 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:04.888624 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-185595da-29888655-crpzq" event={"ID":"1e608f82-1172-468b-9537-7f1841f0488e","Type":"ContainerDied","Data":"727ff9b27d8e9e7e1a84812f80faa9232cc56ad5f49201b3ed23aeec641703d5"} Oct 30 00:15:04.888724 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:04.888684 5001 scope.go:122] "RemoveContainer" containerID="b9b36f9e7f72266306e3dc2c24a0d9069c899c2da49d2d8310f13ad8c4b3bb6a" Oct 30 00:15:04.888859 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:04.888844 5001 scope.go:122] "RemoveContainer" containerID="727ff9b27d8e9e7e1a84812f80faa9232cc56ad5f49201b3ed23aeec641703d5" Oct 30 00:15:04.889064 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:15:04.889045 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-185595da-29888655-crpzq_odh-ai-gateway-infra(1e608f82-1172-468b-9537-7f1841f0488e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-185595da-29888655-crpzq" podUID="1e608f82-1172-468b-9537-7f1841f0488e" Oct 30 00:15:04.890722 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:04.890701 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ffe924e-z9vx8" event={"ID":"7b2c3136-aeca-4974-bc6c-0086f1f2fcda","Type":"ContainerStarted","Data":"6dad701f3db32f43ee7cf1046254c21c4bed9c8a4d68c1bb5461ff809db8ddbb"} Oct 30 00:15:04.890822 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:04.890744 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ffe924e-z9vx8" event={"ID":"7b2c3136-aeca-4974-bc6c-0086f1f2fcda","Type":"ContainerStarted","Data":"c19f9d48570669ceaa3ec824a738c5ddbfd27e1fb13b0edd846433c4e04eb94b"} Oct 30 00:15:05.005645 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:05.005608 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-deda90ae-7kffx"] Oct 30 00:15:05.008813 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:05.008787 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-deda90ae-7kffx"] Oct 30 00:15:05.067268 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:05.067212 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ffe924e-z9vx8" podStartSLOduration=2.067195668 podStartE2EDuration="2.067195668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:15:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:15:05.064462884 +0000 UTC m=+620.269964067" watchObservedRunningTime="2026-10-30 00:15:05.067195668 +0000 UTC m=+620.272696849" Oct 30 00:15:05.899152 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:05.899124 5001 scope.go:122] "RemoveContainer" containerID="727ff9b27d8e9e7e1a84812f80faa9232cc56ad5f49201b3ed23aeec641703d5" Oct 30 00:15:05.899581 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:15:05.899384 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-185595da-29888655-crpzq_odh-ai-gateway-infra(1e608f82-1172-468b-9537-7f1841f0488e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-185595da-29888655-crpzq" podUID="1e608f82-1172-468b-9537-7f1841f0488e" Oct 30 00:15:05.900618 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:05.900594 5001 generic.go:356] "Generic (PLEG): container finished" podID="7b2c3136-aeca-4974-bc6c-0086f1f2fcda" containerID="6dad701f3db32f43ee7cf1046254c21c4bed9c8a4d68c1bb5461ff809db8ddbb" exitCode=0 Oct 30 00:15:05.900732 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:05.900678 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ffe924e-z9vx8" event={"ID":"7b2c3136-aeca-4974-bc6c-0086f1f2fcda","Type":"ContainerDied","Data":"6dad701f3db32f43ee7cf1046254c21c4bed9c8a4d68c1bb5461ff809db8ddbb"} Oct 30 00:15:06.043361 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:06.043338 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-deda90ae-29888655-k45hj" Oct 30 00:15:06.079067 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:06.079017 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdf0d73b-bbdc-4399-b32b-b5d2b887374d-kube-api-access-zdk88\" (UniqueName: \"kubernetes.io/projected/fdf0d73b-bbdc-4399-b32b-b5d2b887374d-kube-api-access-zdk88\") pod \"fdf0d73b-bbdc-4399-b32b-b5d2b887374d\" (UID: \"fdf0d73b-bbdc-4399-b32b-b5d2b887374d\") " Oct 30 00:15:06.079674 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:06.079224 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdf0d73b-bbdc-4399-b32b-b5d2b887374d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf0d73b-bbdc-4399-b32b-b5d2b887374d-maas-api-service-ca\") pod \"fdf0d73b-bbdc-4399-b32b-b5d2b887374d\" (UID: \"fdf0d73b-bbdc-4399-b32b-b5d2b887374d\") " Oct 30 00:15:06.079674 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:06.079580 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf0d73b-bbdc-4399-b32b-b5d2b887374d-maas-api-service-ca" pod "fdf0d73b-bbdc-4399-b32b-b5d2b887374d" (UID: "fdf0d73b-bbdc-4399-b32b-b5d2b887374d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:15:06.079973 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:06.079953 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf0d73b-bbdc-4399-b32b-b5d2b887374d-maas-api-service-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:15:06.081607 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:06.081573 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf0d73b-bbdc-4399-b32b-b5d2b887374d-kube-api-access-zdk88" pod "fdf0d73b-bbdc-4399-b32b-b5d2b887374d" (UID: "fdf0d73b-bbdc-4399-b32b-b5d2b887374d"). InnerVolumeSpecName "kube-api-access-zdk88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:15:06.181326 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:06.181251 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdk88\" (UniqueName: \"kubernetes.io/projected/fdf0d73b-bbdc-4399-b32b-b5d2b887374d-kube-api-access-zdk88\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:15:06.908873 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:06.908845 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-deda90ae-29888655-k45hj" Oct 30 00:15:06.909380 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:06.908887 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-deda90ae-29888655-k45hj" event={"ID":"fdf0d73b-bbdc-4399-b32b-b5d2b887374d","Type":"ContainerDied","Data":"886d141831651b446550cb5778a84b0eb8d231004e44b3cfba7beb3f8dc0c249"} Oct 30 00:15:06.909380 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:06.908979 5001 scope.go:122] "RemoveContainer" containerID="c51097cc18b8f057470234639978a2c5c5c75afeefed0504944763b5fcb593e5" Oct 30 00:15:06.927728 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:06.927702 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aae647ba-86db-430e-9c5a-ed0b8cb22707" path="/var/lib/kubelet/pods/aae647ba-86db-430e-9c5a-ed0b8cb22707/volumes" Oct 30 00:15:06.958881 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:06.958849 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-deda90ae-29888655-k45hj"] Oct 30 00:15:06.968272 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:06.968244 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-deda90ae-29888655-k45hj"] Oct 30 00:15:07.042449 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:07.042362 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-deda90ae-7b4f645874-jjjnn" podUID="70171153-613b-4185-bfaa-1dd277298e44" containerName="payload-processing" containerID="cri-o://25504cfce8d969c37872643d5bac753c936416438dccfc39d29d97ba89054948" gracePeriod=25 Oct 30 00:15:07.137994 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:07.137967 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ffe924e-z9vx8" Oct 30 00:15:07.190673 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:07.190630 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b2c3136-aeca-4974-bc6c-0086f1f2fcda-kube-api-access-vz2dg\" (UniqueName: \"kubernetes.io/projected/7b2c3136-aeca-4974-bc6c-0086f1f2fcda-kube-api-access-vz2dg\") pod \"7b2c3136-aeca-4974-bc6c-0086f1f2fcda\" (UID: \"7b2c3136-aeca-4974-bc6c-0086f1f2fcda\") " Oct 30 00:15:07.190862 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:07.190692 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b2c3136-aeca-4974-bc6c-0086f1f2fcda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b2c3136-aeca-4974-bc6c-0086f1f2fcda-maas-api-openshift-ca\") pod \"7b2c3136-aeca-4974-bc6c-0086f1f2fcda\" (UID: \"7b2c3136-aeca-4974-bc6c-0086f1f2fcda\") " Oct 30 00:15:07.190862 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:07.190831 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b2c3136-aeca-4974-bc6c-0086f1f2fcda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b2c3136-aeca-4974-bc6c-0086f1f2fcda-maas-api-xks-ca\") pod \"7b2c3136-aeca-4974-bc6c-0086f1f2fcda\" (UID: \"7b2c3136-aeca-4974-bc6c-0086f1f2fcda\") " Oct 30 00:15:07.192085 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:07.192027 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b2c3136-aeca-4974-bc6c-0086f1f2fcda-maas-api-openshift-ca" pod "7b2c3136-aeca-4974-bc6c-0086f1f2fcda" (UID: "7b2c3136-aeca-4974-bc6c-0086f1f2fcda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:15:07.193683 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:07.193652 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b2c3136-aeca-4974-bc6c-0086f1f2fcda-maas-api-xks-ca" pod "7b2c3136-aeca-4974-bc6c-0086f1f2fcda" (UID: "7b2c3136-aeca-4974-bc6c-0086f1f2fcda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:15:07.194620 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:07.194595 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b2c3136-aeca-4974-bc6c-0086f1f2fcda-kube-api-access-vz2dg" pod "7b2c3136-aeca-4974-bc6c-0086f1f2fcda" (UID: "7b2c3136-aeca-4974-bc6c-0086f1f2fcda"). InnerVolumeSpecName "kube-api-access-vz2dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:15:07.292313 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:07.292280 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b2c3136-aeca-4974-bc6c-0086f1f2fcda-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:15:07.292313 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:07.292311 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz2dg\" (UniqueName: \"kubernetes.io/projected/7b2c3136-aeca-4974-bc6c-0086f1f2fcda-kube-api-access-vz2dg\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:15:07.292313 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:07.292321 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b2c3136-aeca-4974-bc6c-0086f1f2fcda-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:15:07.316835 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:07.316809 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-deda90ae-7b4f645874-jjjnn" Oct 30 00:15:07.392985 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:07.392947 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70171153-613b-4185-bfaa-1dd277298e44-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70171153-613b-4185-bfaa-1dd277298e44-plugins-config-volume\") pod \"70171153-613b-4185-bfaa-1dd277298e44\" (UID: \"70171153-613b-4185-bfaa-1dd277298e44\") " Oct 30 00:15:07.393167 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:07.393005 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70171153-613b-4185-bfaa-1dd277298e44-kube-api-access-84qx6\" (UniqueName: \"kubernetes.io/projected/70171153-613b-4185-bfaa-1dd277298e44-kube-api-access-84qx6\") pod \"70171153-613b-4185-bfaa-1dd277298e44\" (UID: \"70171153-613b-4185-bfaa-1dd277298e44\") " Oct 30 00:15:07.393396 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:07.393372 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70171153-613b-4185-bfaa-1dd277298e44-plugins-config-volume" pod "70171153-613b-4185-bfaa-1dd277298e44" (UID: "70171153-613b-4185-bfaa-1dd277298e44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:15:07.395176 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:07.395151 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70171153-613b-4185-bfaa-1dd277298e44-kube-api-access-84qx6" pod "70171153-613b-4185-bfaa-1dd277298e44" (UID: "70171153-613b-4185-bfaa-1dd277298e44"). InnerVolumeSpecName "kube-api-access-84qx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:15:07.493933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:07.493835 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70171153-613b-4185-bfaa-1dd277298e44-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:15:07.493933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:07.493868 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84qx6\" (UniqueName: \"kubernetes.io/projected/70171153-613b-4185-bfaa-1dd277298e44-kube-api-access-84qx6\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:15:07.895242 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:07.895206 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:15:07.908256 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:07.908226 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:15:07.915873 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:07.915847 5001 generic.go:356] "Generic (PLEG): container finished" podID="70171153-613b-4185-bfaa-1dd277298e44" containerID="25504cfce8d969c37872643d5bac753c936416438dccfc39d29d97ba89054948" exitCode=0 Oct 30 00:15:07.916297 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:07.915890 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-deda90ae-7b4f645874-jjjnn" Oct 30 00:15:07.916297 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:07.915933 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-deda90ae-7b4f645874-jjjnn" event={"ID":"70171153-613b-4185-bfaa-1dd277298e44","Type":"ContainerDied","Data":"25504cfce8d969c37872643d5bac753c936416438dccfc39d29d97ba89054948"} Oct 30 00:15:07.916297 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:07.915981 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-deda90ae-7b4f645874-jjjnn" event={"ID":"70171153-613b-4185-bfaa-1dd277298e44","Type":"ContainerDied","Data":"109730bd7e784391b8e9170155dfbc8fa95605e7fd691d4abd1531795ecb9de6"} Oct 30 00:15:07.916297 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:07.916010 5001 scope.go:122] "RemoveContainer" containerID="25504cfce8d969c37872643d5bac753c936416438dccfc39d29d97ba89054948" Oct 30 00:15:07.921290 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:07.921272 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ffe924e-z9vx8" Oct 30 00:15:07.921403 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:07.921310 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ffe924e-z9vx8" event={"ID":"7b2c3136-aeca-4974-bc6c-0086f1f2fcda","Type":"ContainerDied","Data":"c19f9d48570669ceaa3ec824a738c5ddbfd27e1fb13b0edd846433c4e04eb94b"} Oct 30 00:15:07.921403 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:07.921351 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c19f9d48570669ceaa3ec824a738c5ddbfd27e1fb13b0edd846433c4e04eb94b" Oct 30 00:15:07.926874 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:07.926858 5001 scope.go:122] "RemoveContainer" containerID="25504cfce8d969c37872643d5bac753c936416438dccfc39d29d97ba89054948" Oct 30 00:15:07.927183 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:15:07.927163 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25504cfce8d969c37872643d5bac753c936416438dccfc39d29d97ba89054948\": container with ID starting with 25504cfce8d969c37872643d5bac753c936416438dccfc39d29d97ba89054948 not found: ID does not exist" containerID="25504cfce8d969c37872643d5bac753c936416438dccfc39d29d97ba89054948" Oct 30 00:15:07.927252 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:07.927194 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25504cfce8d969c37872643d5bac753c936416438dccfc39d29d97ba89054948"} err="failed to get container status \"25504cfce8d969c37872643d5bac753c936416438dccfc39d29d97ba89054948\": rpc error: code = NotFound desc = could not find container \"25504cfce8d969c37872643d5bac753c936416438dccfc39d29d97ba89054948\": container with ID starting with 25504cfce8d969c37872643d5bac753c936416438dccfc39d29d97ba89054948 not found: ID does not exist" Oct 30 00:15:07.935900 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:07.935884 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:15:08.023196 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:08.023165 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-deda90ae-7b4f645874-jjjnn"] Oct 30 00:15:08.027960 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:08.027901 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-deda90ae-7b4f645874-jjjnn"] Oct 30 00:15:08.840788 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:08.840747 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-185595da-6f9d6ddd86-6cfxr" podUID="e76cf1c1-30bf-4823-a7c5-53380f123448" containerName="payload-processing" containerID="cri-o://74b834185e32b10d6c4b5b5b52b7792bebe0b438b280ce5e511ce938b207401b" gracePeriod=25 Oct 30 00:15:08.917694 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:08.917662 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70171153-613b-4185-bfaa-1dd277298e44" path="/var/lib/kubelet/pods/70171153-613b-4185-bfaa-1dd277298e44/volumes" Oct 30 00:15:08.918056 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:08.918032 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdf0d73b-bbdc-4399-b32b-b5d2b887374d" path="/var/lib/kubelet/pods/fdf0d73b-bbdc-4399-b32b-b5d2b887374d/volumes" Oct 30 00:15:09.135251 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:09.135228 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-185595da-6f9d6ddd86-6cfxr" Oct 30 00:15:09.212852 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:09.212820 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e76cf1c1-30bf-4823-a7c5-53380f123448-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76cf1c1-30bf-4823-a7c5-53380f123448-plugins-config-volume\") pod \"e76cf1c1-30bf-4823-a7c5-53380f123448\" (UID: \"e76cf1c1-30bf-4823-a7c5-53380f123448\") " Oct 30 00:15:09.213037 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:09.212863 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e76cf1c1-30bf-4823-a7c5-53380f123448-kube-api-access-mnj4f\" (UniqueName: \"kubernetes.io/projected/e76cf1c1-30bf-4823-a7c5-53380f123448-kube-api-access-mnj4f\") pod \"e76cf1c1-30bf-4823-a7c5-53380f123448\" (UID: \"e76cf1c1-30bf-4823-a7c5-53380f123448\") " Oct 30 00:15:09.213235 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:09.213210 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e76cf1c1-30bf-4823-a7c5-53380f123448-plugins-config-volume" pod "e76cf1c1-30bf-4823-a7c5-53380f123448" (UID: "e76cf1c1-30bf-4823-a7c5-53380f123448"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:15:09.214992 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:09.214968 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e76cf1c1-30bf-4823-a7c5-53380f123448-kube-api-access-mnj4f" pod "e76cf1c1-30bf-4823-a7c5-53380f123448" (UID: "e76cf1c1-30bf-4823-a7c5-53380f123448"). InnerVolumeSpecName "kube-api-access-mnj4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:15:09.313774 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:09.313732 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76cf1c1-30bf-4823-a7c5-53380f123448-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:15:09.313774 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:09.313774 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnj4f\" (UniqueName: \"kubernetes.io/projected/e76cf1c1-30bf-4823-a7c5-53380f123448-kube-api-access-mnj4f\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:15:09.935812 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:09.935768 5001 generic.go:356] "Generic (PLEG): container finished" podID="e76cf1c1-30bf-4823-a7c5-53380f123448" containerID="74b834185e32b10d6c4b5b5b52b7792bebe0b438b280ce5e511ce938b207401b" exitCode=0 Oct 30 00:15:09.935812 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:09.935811 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-185595da-6f9d6ddd86-6cfxr" Oct 30 00:15:09.936355 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:09.935861 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-185595da-6f9d6ddd86-6cfxr" event={"ID":"e76cf1c1-30bf-4823-a7c5-53380f123448","Type":"ContainerDied","Data":"74b834185e32b10d6c4b5b5b52b7792bebe0b438b280ce5e511ce938b207401b"} Oct 30 00:15:09.936355 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:09.935950 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-185595da-6f9d6ddd86-6cfxr" event={"ID":"e76cf1c1-30bf-4823-a7c5-53380f123448","Type":"ContainerDied","Data":"7c3868c3fc5b83badaef7b92e3063587122afabdd3e7755681559025887a0a27"} Oct 30 00:15:09.936355 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:09.935986 5001 scope.go:122] "RemoveContainer" containerID="74b834185e32b10d6c4b5b5b52b7792bebe0b438b280ce5e511ce938b207401b" Oct 30 00:15:09.946420 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:09.946402 5001 scope.go:122] "RemoveContainer" containerID="74b834185e32b10d6c4b5b5b52b7792bebe0b438b280ce5e511ce938b207401b" Oct 30 00:15:09.946709 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:15:09.946688 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74b834185e32b10d6c4b5b5b52b7792bebe0b438b280ce5e511ce938b207401b\": container with ID starting with 74b834185e32b10d6c4b5b5b52b7792bebe0b438b280ce5e511ce938b207401b not found: ID does not exist" containerID="74b834185e32b10d6c4b5b5b52b7792bebe0b438b280ce5e511ce938b207401b" Oct 30 00:15:09.946805 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:09.946716 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74b834185e32b10d6c4b5b5b52b7792bebe0b438b280ce5e511ce938b207401b"} err="failed to get container status \"74b834185e32b10d6c4b5b5b52b7792bebe0b438b280ce5e511ce938b207401b\": rpc error: code = NotFound desc = could not find container \"74b834185e32b10d6c4b5b5b52b7792bebe0b438b280ce5e511ce938b207401b\": container with ID starting with 74b834185e32b10d6c4b5b5b52b7792bebe0b438b280ce5e511ce938b207401b not found: ID does not exist" Oct 30 00:15:09.987172 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:09.987125 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-185595da-6f9d6ddd86-6cfxr"] Oct 30 00:15:09.990756 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:09.990730 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-185595da-6f9d6ddd86-6cfxr"] Oct 30 00:15:10.764747 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.764707 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp"] Oct 30 00:15:10.765446 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.765424 5001 cpu_manager.go:414] "containerMap: removing container" podUID="fdf0d73b-bbdc-4399-b32b-b5d2b887374d" containerName="cleanup" Oct 30 00:15:10.765531 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.765449 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdf0d73b-bbdc-4399-b32b-b5d2b887374d" containerName="cleanup" Oct 30 00:15:10.765531 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.765492 5001 cpu_manager.go:414] "containerMap: removing container" podUID="70171153-613b-4185-bfaa-1dd277298e44" containerName="payload-processing" Oct 30 00:15:10.765531 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.765500 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70171153-613b-4185-bfaa-1dd277298e44" containerName="payload-processing" Oct 30 00:15:10.765531 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.765517 5001 cpu_manager.go:414] "containerMap: removing container" podUID="7b2c3136-aeca-4974-bc6c-0086f1f2fcda" containerName="revoke-keys" Oct 30 00:15:10.765531 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.765525 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b2c3136-aeca-4974-bc6c-0086f1f2fcda" containerName="revoke-keys" Oct 30 00:15:10.765786 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.765544 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e76cf1c1-30bf-4823-a7c5-53380f123448" containerName="payload-processing" Oct 30 00:15:10.765786 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.765552 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e76cf1c1-30bf-4823-a7c5-53380f123448" containerName="payload-processing" Oct 30 00:15:10.765786 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.765676 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdf0d73b-bbdc-4399-b32b-b5d2b887374d" containerName="cleanup" Oct 30 00:15:10.765786 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.765692 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="70171153-613b-4185-bfaa-1dd277298e44" containerName="payload-processing" Oct 30 00:15:10.765786 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.765704 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b2c3136-aeca-4974-bc6c-0086f1f2fcda" containerName="revoke-keys" Oct 30 00:15:10.765786 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.765714 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e76cf1c1-30bf-4823-a7c5-53380f123448" containerName="payload-processing" Oct 30 00:15:10.775174 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.775146 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:15:10.786429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.786363 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp"] Oct 30 00:15:10.792235 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.792201 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-6c45bb\"/\"openshift-service-ca.crt\"" Oct 30 00:15:10.792235 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.792201 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-6c45bb\"/\"kube-root-ca.crt\"" Oct 30 00:15:10.792435 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.792239 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-6c45bb\"/\"default-dockercfg-z7289\"" Oct 30 00:15:10.798184 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.798162 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-6c45bb\"/\"simulator-w6-6c45bb-kserve-self-signed-certs\"" Oct 30 00:15:10.826282 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.826251 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0c7951f-8758-4b9c-b91a-00024f9bf582-dshm\") pod \"simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp\" (UID: \"b0c7951f-8758-4b9c-b91a-00024f9bf582\") " pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:15:10.826446 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.826299 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0c7951f-8758-4b9c-b91a-00024f9bf582-tmp-dir\") pod \"simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp\" (UID: \"b0c7951f-8758-4b9c-b91a-00024f9bf582\") " pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:15:10.826446 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.826368 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0c7951f-8758-4b9c-b91a-00024f9bf582-home\") pod \"simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp\" (UID: \"b0c7951f-8758-4b9c-b91a-00024f9bf582\") " pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:15:10.826569 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.826500 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0c7951f-8758-4b9c-b91a-00024f9bf582-model-cache\") pod \"simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp\" (UID: \"b0c7951f-8758-4b9c-b91a-00024f9bf582\") " pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:15:10.826569 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.826554 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g26qv\" (UniqueName: \"kubernetes.io/projected/b0c7951f-8758-4b9c-b91a-00024f9bf582-kube-api-access-g26qv\") pod \"simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp\" (UID: \"b0c7951f-8758-4b9c-b91a-00024f9bf582\") " pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:15:10.826643 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.826604 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c7951f-8758-4b9c-b91a-00024f9bf582-tls-certs\") pod \"simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp\" (UID: \"b0c7951f-8758-4b9c-b91a-00024f9bf582\") " pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:15:10.917404 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.917369 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e76cf1c1-30bf-4823-a7c5-53380f123448" path="/var/lib/kubelet/pods/e76cf1c1-30bf-4823-a7c5-53380f123448/volumes" Oct 30 00:15:10.927169 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.927135 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0c7951f-8758-4b9c-b91a-00024f9bf582-dshm\") pod \"simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp\" (UID: \"b0c7951f-8758-4b9c-b91a-00024f9bf582\") " pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:15:10.927328 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.927179 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0c7951f-8758-4b9c-b91a-00024f9bf582-tmp-dir\") pod \"simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp\" (UID: \"b0c7951f-8758-4b9c-b91a-00024f9bf582\") " pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:15:10.927328 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.927285 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0c7951f-8758-4b9c-b91a-00024f9bf582-home\") pod \"simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp\" (UID: \"b0c7951f-8758-4b9c-b91a-00024f9bf582\") " pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:15:10.927448 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.927353 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0c7951f-8758-4b9c-b91a-00024f9bf582-model-cache\") pod \"simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp\" (UID: \"b0c7951f-8758-4b9c-b91a-00024f9bf582\") " pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:15:10.927502 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.927444 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g26qv\" (UniqueName: \"kubernetes.io/projected/b0c7951f-8758-4b9c-b91a-00024f9bf582-kube-api-access-g26qv\") pod \"simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp\" (UID: \"b0c7951f-8758-4b9c-b91a-00024f9bf582\") " pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:15:10.927560 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.927510 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c7951f-8758-4b9c-b91a-00024f9bf582-tls-certs\") pod \"simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp\" (UID: \"b0c7951f-8758-4b9c-b91a-00024f9bf582\") " pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:15:10.927560 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.927520 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0c7951f-8758-4b9c-b91a-00024f9bf582-tmp-dir\") pod \"simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp\" (UID: \"b0c7951f-8758-4b9c-b91a-00024f9bf582\") " pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:15:10.927667 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.927638 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0c7951f-8758-4b9c-b91a-00024f9bf582-home\") pod \"simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp\" (UID: \"b0c7951f-8758-4b9c-b91a-00024f9bf582\") " pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:15:10.927957 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.927933 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0c7951f-8758-4b9c-b91a-00024f9bf582-model-cache\") pod \"simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp\" (UID: \"b0c7951f-8758-4b9c-b91a-00024f9bf582\") " pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:15:10.929551 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.929525 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0c7951f-8758-4b9c-b91a-00024f9bf582-dshm\") pod \"simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp\" (UID: \"b0c7951f-8758-4b9c-b91a-00024f9bf582\") " pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:15:10.929995 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.929981 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c7951f-8758-4b9c-b91a-00024f9bf582-tls-certs\") pod \"simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp\" (UID: \"b0c7951f-8758-4b9c-b91a-00024f9bf582\") " pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:15:10.948253 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:10.948231 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g26qv\" (UniqueName: \"kubernetes.io/projected/b0c7951f-8758-4b9c-b91a-00024f9bf582-kube-api-access-g26qv\") pod \"simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp\" (UID: \"b0c7951f-8758-4b9c-b91a-00024f9bf582\") " pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:15:11.091840 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:11.091744 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:15:11.390566 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:11.390536 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp"] Oct 30 00:15:11.393198 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:15:11.393171 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0c7951f_8758_4b9c_b91a_00024f9bf582.slice/crio-a2e0834bac1fa769b4eb430be3fe17de567d7f79a7d7b71ffb32ce3fcaa47d6b WatchSource:0}: Error finding container a2e0834bac1fa769b4eb430be3fe17de567d7f79a7d7b71ffb32ce3fcaa47d6b: Status 404 returned error can't find the container with id a2e0834bac1fa769b4eb430be3fe17de567d7f79a7d7b71ffb32ce3fcaa47d6b Oct 30 00:15:11.964717 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:11.964680 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" event={"ID":"b0c7951f-8758-4b9c-b91a-00024f9bf582","Type":"ContainerStarted","Data":"a262e489e843f3476800ff87eac5543a3377c42dd3509865ef8b2c49c30f7201"} Oct 30 00:15:11.965339 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:11.964745 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" event={"ID":"b0c7951f-8758-4b9c-b91a-00024f9bf582","Type":"ContainerStarted","Data":"a2e0834bac1fa769b4eb430be3fe17de567d7f79a7d7b71ffb32ce3fcaa47d6b"} Oct 30 00:15:12.037217 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:12.037165 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podStartSLOduration=2.037151627 podStartE2EDuration="2.037151627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:15:12.03463559 +0000 UTC m=+627.240136784" watchObservedRunningTime="2026-10-30 00:15:12.037151627 +0000 UTC m=+627.242652856" Oct 30 00:15:12.092905 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:12.092871 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:15:13.805940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:13.805895 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8ffe924e-5fd58cdcf8-pxzcm"] Oct 30 00:15:13.911606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:13.911419 5001 scope.go:122] "RemoveContainer" containerID="12fb01a0d1d1cc4b8b59bbb0aff4de8b9fd1089be454039eb470667845cfae5b" Oct 30 00:15:13.911786 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:13.911624 5001 scope.go:122] "RemoveContainer" containerID="246aca4957b0fe7e6931f3cee857c7992c11f6a1c57729ea9901e60248c8d1da" Oct 30 00:15:14.821845 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:14.821815 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-185595da-29888655-crpzq"] Oct 30 00:15:15.006670 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:15.006620 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-185595da-29888655-crpzq" Oct 30 00:15:15.012255 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:15.012226 5001 generic.go:356] "Generic (PLEG): container finished" podID="75ca90e5-c780-4851-92e0-f3619116c5f3" containerID="f8d0894c30dbd43a7565896a579a012b9455853b9dec772aa835bee23bd06efa" exitCode=60 Oct 30 00:15:15.012481 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:15.012263 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89" event={"ID":"75ca90e5-c780-4851-92e0-f3619116c5f3","Type":"ContainerDied","Data":"f8d0894c30dbd43a7565896a579a012b9455853b9dec772aa835bee23bd06efa"} Oct 30 00:15:15.012481 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:15.012331 5001 scope.go:122] "RemoveContainer" containerID="246aca4957b0fe7e6931f3cee857c7992c11f6a1c57729ea9901e60248c8d1da" Oct 30 00:15:15.012793 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:15.012775 5001 scope.go:122] "RemoveContainer" containerID="f8d0894c30dbd43a7565896a579a012b9455853b9dec772aa835bee23bd06efa" Oct 30 00:15:15.013143 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:15:15.013104 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89_odh-ai-gateway-infra(75ca90e5-c780-4851-92e0-f3619116c5f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89" podUID="75ca90e5-c780-4851-92e0-f3619116c5f3" Oct 30 00:15:15.015361 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:15.015336 5001 generic.go:356] "Generic (PLEG): container finished" podID="7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d" containerID="dfa4ec872f80ef35d2dc6471f539e704c272025a7a0763869538d7c44051cbbc" exitCode=60 Oct 30 00:15:15.015472 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:15.015372 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6" event={"ID":"7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d","Type":"ContainerDied","Data":"dfa4ec872f80ef35d2dc6471f539e704c272025a7a0763869538d7c44051cbbc"} Oct 30 00:15:15.015764 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:15.015745 5001 scope.go:122] "RemoveContainer" containerID="dfa4ec872f80ef35d2dc6471f539e704c272025a7a0763869538d7c44051cbbc" Oct 30 00:15:15.018617 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:15:15.016015 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6_odh-ai-gateway-infra(7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6" podUID="7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d" Oct 30 00:15:15.018617 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:15.017600 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-185595da-29888655-crpzq" event={"ID":"1e608f82-1172-468b-9537-7f1841f0488e","Type":"ContainerDied","Data":"f5b1f7fb582c3b6ddbe57ace1ea6f0597bea9a49efc82c824c55988b21d5c9db"} Oct 30 00:15:15.018617 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:15.017690 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-185595da-29888655-crpzq" Oct 30 00:15:15.028670 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:15.028649 5001 scope.go:122] "RemoveContainer" containerID="12fb01a0d1d1cc4b8b59bbb0aff4de8b9fd1089be454039eb470667845cfae5b" Oct 30 00:15:15.042615 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:15.042595 5001 scope.go:122] "RemoveContainer" containerID="727ff9b27d8e9e7e1a84812f80faa9232cc56ad5f49201b3ed23aeec641703d5" Oct 30 00:15:15.073137 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:15.073107 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e608f82-1172-468b-9537-7f1841f0488e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e608f82-1172-468b-9537-7f1841f0488e-maas-api-service-ca\") pod \"1e608f82-1172-468b-9537-7f1841f0488e\" (UID: \"1e608f82-1172-468b-9537-7f1841f0488e\") " Oct 30 00:15:15.073278 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:15.073205 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e608f82-1172-468b-9537-7f1841f0488e-kube-api-access-sr6z2\" (UniqueName: \"kubernetes.io/projected/1e608f82-1172-468b-9537-7f1841f0488e-kube-api-access-sr6z2\") pod \"1e608f82-1172-468b-9537-7f1841f0488e\" (UID: \"1e608f82-1172-468b-9537-7f1841f0488e\") " Oct 30 00:15:15.073562 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:15.073536 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e608f82-1172-468b-9537-7f1841f0488e-maas-api-service-ca" pod "1e608f82-1172-468b-9537-7f1841f0488e" (UID: "1e608f82-1172-468b-9537-7f1841f0488e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:15:15.075526 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:15.075503 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e608f82-1172-468b-9537-7f1841f0488e-kube-api-access-sr6z2" pod "1e608f82-1172-468b-9537-7f1841f0488e" (UID: "1e608f82-1172-468b-9537-7f1841f0488e"). InnerVolumeSpecName "kube-api-access-sr6z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:15:15.174114 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:15.174084 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e608f82-1172-468b-9537-7f1841f0488e-maas-api-service-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:15:15.174114 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:15.174112 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr6z2\" (UniqueName: \"kubernetes.io/projected/1e608f82-1172-468b-9537-7f1841f0488e-kube-api-access-sr6z2\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:15:15.359145 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:15.359087 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-185595da-29888655-crpzq"] Oct 30 00:15:15.363541 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:15.363516 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-185595da-29888655-crpzq"] Oct 30 00:15:15.422399 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:15.422368 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-185595da-xq2pp"] Oct 30 00:15:15.425016 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:15.424994 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-185595da-xq2pp"] Oct 30 00:15:16.093852 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:16.093823 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89"] Oct 30 00:15:16.153372 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:16.153339 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6"] Oct 30 00:15:16.245178 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:16.245156 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89" Oct 30 00:15:16.286636 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:16.286604 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75ca90e5-c780-4851-92e0-f3619116c5f3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75ca90e5-c780-4851-92e0-f3619116c5f3-maas-api-service-ca\") pod \"75ca90e5-c780-4851-92e0-f3619116c5f3\" (UID: \"75ca90e5-c780-4851-92e0-f3619116c5f3\") " Oct 30 00:15:16.286794 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:16.286730 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75ca90e5-c780-4851-92e0-f3619116c5f3-kube-api-access-vvzt5\" (UniqueName: \"kubernetes.io/projected/75ca90e5-c780-4851-92e0-f3619116c5f3-kube-api-access-vvzt5\") pod \"75ca90e5-c780-4851-92e0-f3619116c5f3\" (UID: \"75ca90e5-c780-4851-92e0-f3619116c5f3\") " Oct 30 00:15:16.287066 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:16.287037 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75ca90e5-c780-4851-92e0-f3619116c5f3-maas-api-service-ca" pod "75ca90e5-c780-4851-92e0-f3619116c5f3" (UID: "75ca90e5-c780-4851-92e0-f3619116c5f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:15:16.289053 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:16.289030 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ca90e5-c780-4851-92e0-f3619116c5f3-kube-api-access-vvzt5" pod "75ca90e5-c780-4851-92e0-f3619116c5f3" (UID: "75ca90e5-c780-4851-92e0-f3619116c5f3"). InnerVolumeSpecName "kube-api-access-vvzt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:15:16.314078 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:16.314057 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6" Oct 30 00:15:16.388303 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:16.388235 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d-maas-api-service-ca\") pod \"7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d\" (UID: \"7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d\") " Oct 30 00:15:16.388303 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:16.388291 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d-kube-api-access-x9jwd\" (UniqueName: \"kubernetes.io/projected/7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d-kube-api-access-x9jwd\") pod \"7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d\" (UID: \"7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d\") " Oct 30 00:15:16.388476 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:16.388459 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvzt5\" (UniqueName: \"kubernetes.io/projected/75ca90e5-c780-4851-92e0-f3619116c5f3-kube-api-access-vvzt5\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:15:16.388476 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:16.388469 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75ca90e5-c780-4851-92e0-f3619116c5f3-maas-api-service-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:15:16.388591 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:16.388569 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d-maas-api-service-ca" pod "7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d" (UID: "7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:15:16.390327 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:16.390306 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d-kube-api-access-x9jwd" pod "7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d" (UID: "7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d"). InnerVolumeSpecName "kube-api-access-x9jwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:15:16.489138 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:16.489105 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d-maas-api-service-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:15:16.489138 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:16.489135 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9jwd\" (UniqueName: \"kubernetes.io/projected/7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d-kube-api-access-x9jwd\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:15:16.920699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:16.920663 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e608f82-1172-468b-9537-7f1841f0488e" path="/var/lib/kubelet/pods/1e608f82-1172-468b-9537-7f1841f0488e/volumes" Oct 30 00:15:16.921476 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:16.921450 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31e047e4-bc04-4d68-945e-39251841a051" path="/var/lib/kubelet/pods/31e047e4-bc04-4d68-945e-39251841a051/volumes" Oct 30 00:15:17.043201 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:17.037751 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89" event={"ID":"75ca90e5-c780-4851-92e0-f3619116c5f3","Type":"ContainerDied","Data":"681d7b514fefb9d51d7774196d01d02fa99b13b34212c73d5b46c16b58a2be9c"} Oct 30 00:15:17.043201 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:17.037821 5001 scope.go:122] "RemoveContainer" containerID="f8d0894c30dbd43a7565896a579a012b9455853b9dec772aa835bee23bd06efa" Oct 30 00:15:17.043201 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:17.037972 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89" Oct 30 00:15:17.043201 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:17.041614 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6" event={"ID":"7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d","Type":"ContainerDied","Data":"0219ed2d81aa3cf7f85a85f8a3f4b4fcbdad41810d26ed7c27c3f63b393f2601"} Oct 30 00:15:17.043201 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:17.041713 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6" Oct 30 00:15:17.058932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:17.058897 5001 scope.go:122] "RemoveContainer" containerID="dfa4ec872f80ef35d2dc6471f539e704c272025a7a0763869538d7c44051cbbc" Oct 30 00:15:17.083709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:17.083681 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89"] Oct 30 00:15:17.094719 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:17.094690 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888655-2vf89"] Oct 30 00:15:17.125270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:17.125237 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6"] Oct 30 00:15:17.129516 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:17.129491 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6c45bb-29888655-jp4x6"] Oct 30 00:15:17.321268 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:15:17.321231 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:15:17.584165 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:17.584080 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c9e9fa7-9lrln"] Oct 30 00:15:17.586631 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:17.586606 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c9e9fa7-9lrln"] Oct 30 00:15:18.806806 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:18.806763 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-8ffe924e-5fd58cdcf8-pxzcm" podUID="336500a6-7517-4793-9e4c-64ea0e676fc6" containerName="payload-processing" containerID="cri-o://82144cf03c0833c90eb119cb64398efc7795de7c2e033cdd8848b1bd6ad2efb6" gracePeriod=25 Oct 30 00:15:18.919096 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:18.919065 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75ca90e5-c780-4851-92e0-f3619116c5f3" path="/var/lib/kubelet/pods/75ca90e5-c780-4851-92e0-f3619116c5f3/volumes" Oct 30 00:15:18.919459 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:18.919446 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d" path="/var/lib/kubelet/pods/7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d/volumes" Oct 30 00:15:18.919807 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:18.919796 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b311c78-77ca-485b-a3e5-5dd368bbb94b" path="/var/lib/kubelet/pods/8b311c78-77ca-485b-a3e5-5dd368bbb94b/volumes" Oct 30 00:15:19.057628 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:19.057547 5001 generic.go:356] "Generic (PLEG): container finished" podID="336500a6-7517-4793-9e4c-64ea0e676fc6" containerID="82144cf03c0833c90eb119cb64398efc7795de7c2e033cdd8848b1bd6ad2efb6" exitCode=0 Oct 30 00:15:19.057884 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:19.057621 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8ffe924e-5fd58cdcf8-pxzcm" event={"ID":"336500a6-7517-4793-9e4c-64ea0e676fc6","Type":"ContainerDied","Data":"82144cf03c0833c90eb119cb64398efc7795de7c2e033cdd8848b1bd6ad2efb6"} Oct 30 00:15:19.079257 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:19.079227 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8ffe924e-5fd58cdcf8-pxzcm" Oct 30 00:15:19.123438 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:19.123401 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/336500a6-7517-4793-9e4c-64ea0e676fc6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/336500a6-7517-4793-9e4c-64ea0e676fc6-plugins-config-volume\") pod \"336500a6-7517-4793-9e4c-64ea0e676fc6\" (UID: \"336500a6-7517-4793-9e4c-64ea0e676fc6\") " Oct 30 00:15:19.123600 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:19.123549 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/336500a6-7517-4793-9e4c-64ea0e676fc6-kube-api-access-vpf6s\" (UniqueName: \"kubernetes.io/projected/336500a6-7517-4793-9e4c-64ea0e676fc6-kube-api-access-vpf6s\") pod \"336500a6-7517-4793-9e4c-64ea0e676fc6\" (UID: \"336500a6-7517-4793-9e4c-64ea0e676fc6\") " Oct 30 00:15:19.129006 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:19.126323 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/336500a6-7517-4793-9e4c-64ea0e676fc6-plugins-config-volume" pod "336500a6-7517-4793-9e4c-64ea0e676fc6" (UID: "336500a6-7517-4793-9e4c-64ea0e676fc6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:15:19.129950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:19.129896 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/336500a6-7517-4793-9e4c-64ea0e676fc6-kube-api-access-vpf6s" pod "336500a6-7517-4793-9e4c-64ea0e676fc6" (UID: "336500a6-7517-4793-9e4c-64ea0e676fc6"). InnerVolumeSpecName "kube-api-access-vpf6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:15:19.225068 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:19.225034 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpf6s\" (UniqueName: \"kubernetes.io/projected/336500a6-7517-4793-9e4c-64ea0e676fc6-kube-api-access-vpf6s\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:15:19.225068 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:19.225067 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/336500a6-7517-4793-9e4c-64ea0e676fc6-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:15:20.067113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:20.067084 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8ffe924e-5fd58cdcf8-pxzcm" Oct 30 00:15:20.067606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:20.067124 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8ffe924e-5fd58cdcf8-pxzcm" event={"ID":"336500a6-7517-4793-9e4c-64ea0e676fc6","Type":"ContainerDied","Data":"6f45ebe821867809562c40929ab14544585317c8b951256ad0dcc791fad577c8"} Oct 30 00:15:20.067606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:20.067188 5001 scope.go:122] "RemoveContainer" containerID="82144cf03c0833c90eb119cb64398efc7795de7c2e033cdd8848b1bd6ad2efb6" Oct 30 00:15:20.111408 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:20.111376 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8ffe924e-5fd58cdcf8-pxzcm"] Oct 30 00:15:20.121064 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:20.121036 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8ffe924e-5fd58cdcf8-pxzcm"] Oct 30 00:15:20.916891 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:20.916857 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="336500a6-7517-4793-9e4c-64ea0e676fc6" path="/var/lib/kubelet/pods/336500a6-7517-4793-9e4c-64ea0e676fc6/volumes" Oct 30 00:15:21.092089 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:21.092059 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:15:21.104968 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:21.104943 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:15:22.092907 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:22.092878 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:15:23.156120 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:23.156083 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ffe924e-z9vx8"] Oct 30 00:15:23.159833 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:23.159803 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ffe924e-z9vx8"] Oct 30 00:15:24.516967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.516906 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-02665b-c768b5f9b-dpvxb"] Oct 30 00:15:24.518134 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.518106 5001 cpu_manager.go:414] "containerMap: removing container" podUID="336500a6-7517-4793-9e4c-64ea0e676fc6" containerName="payload-processing" Oct 30 00:15:24.518297 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.518281 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="336500a6-7517-4793-9e4c-64ea0e676fc6" containerName="payload-processing" Oct 30 00:15:24.518379 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.518369 5001 cpu_manager.go:414] "containerMap: removing container" podUID="7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d" containerName="cleanup" Oct 30 00:15:24.518509 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.518495 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d" containerName="cleanup" Oct 30 00:15:24.518603 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.518591 5001 cpu_manager.go:414] "containerMap: removing container" podUID="75ca90e5-c780-4851-92e0-f3619116c5f3" containerName="cleanup" Oct 30 00:15:24.518686 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.518674 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75ca90e5-c780-4851-92e0-f3619116c5f3" containerName="cleanup" Oct 30 00:15:24.518782 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.518772 5001 cpu_manager.go:414] "containerMap: removing container" podUID="7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d" containerName="cleanup" Oct 30 00:15:24.518857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.518847 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d" containerName="cleanup" Oct 30 00:15:24.518952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.518941 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1e608f82-1172-468b-9537-7f1841f0488e" containerName="cleanup" Oct 30 00:15:24.519058 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.519038 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e608f82-1172-468b-9537-7f1841f0488e" containerName="cleanup" Oct 30 00:15:24.519136 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.519127 5001 cpu_manager.go:414] "containerMap: removing container" podUID="7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d" containerName="cleanup" Oct 30 00:15:24.519207 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.519196 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d" containerName="cleanup" Oct 30 00:15:24.519275 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.519266 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1e608f82-1172-468b-9537-7f1841f0488e" containerName="cleanup" Oct 30 00:15:24.519347 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.519337 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e608f82-1172-468b-9537-7f1841f0488e" containerName="cleanup" Oct 30 00:15:24.519455 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.519444 5001 cpu_manager.go:414] "containerMap: removing container" podUID="75ca90e5-c780-4851-92e0-f3619116c5f3" containerName="cleanup" Oct 30 00:15:24.519538 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.519526 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75ca90e5-c780-4851-92e0-f3619116c5f3" containerName="cleanup" Oct 30 00:15:24.519613 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.519604 5001 cpu_manager.go:414] "containerMap: removing container" podUID="fdf0d73b-bbdc-4399-b32b-b5d2b887374d" containerName="cleanup" Oct 30 00:15:24.519682 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.519672 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdf0d73b-bbdc-4399-b32b-b5d2b887374d" containerName="cleanup" Oct 30 00:15:24.519866 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.519854 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="75ca90e5-c780-4851-92e0-f3619116c5f3" containerName="cleanup" Oct 30 00:15:24.520000 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.519987 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdf0d73b-bbdc-4399-b32b-b5d2b887374d" containerName="cleanup" Oct 30 00:15:24.520077 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.520068 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="336500a6-7517-4793-9e4c-64ea0e676fc6" containerName="payload-processing" Oct 30 00:15:24.520153 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.520144 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d" containerName="cleanup" Oct 30 00:15:24.520217 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.520209 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="75ca90e5-c780-4851-92e0-f3619116c5f3" containerName="cleanup" Oct 30 00:15:24.520288 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.520279 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d" containerName="cleanup" Oct 30 00:15:24.520357 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.520349 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e608f82-1172-468b-9537-7f1841f0488e" containerName="cleanup" Oct 30 00:15:24.520429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.520420 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cb42483-aef9-4c39-a4f7-0e6a9b7c6e8d" containerName="cleanup" Oct 30 00:15:24.520496 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.520487 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e608f82-1172-468b-9537-7f1841f0488e" containerName="cleanup" Oct 30 00:15:24.525938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.525900 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-02665b-c768b5f9b-dpvxb" Oct 30 00:15:24.541264 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.541238 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-02665b-c768b5f9b-dpvxb"] Oct 30 00:15:24.543090 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.543059 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-02665b-dockercfg-wrxlq\"" Oct 30 00:15:24.543221 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.543060 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-02665b\"" Oct 30 00:15:24.600200 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.600162 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ebb197-56b8-4043-be23-682f002391b5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-02665b-c768b5f9b-dpvxb\" (UID: \"a1ebb197-56b8-4043-be23-682f002391b5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-02665b-c768b5f9b-dpvxb" Oct 30 00:15:24.600344 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.600225 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9pc9\" (UniqueName: \"kubernetes.io/projected/a1ebb197-56b8-4043-be23-682f002391b5-kube-api-access-t9pc9\") pod \"payload-processing-e2e-worker-w4-02665b-c768b5f9b-dpvxb\" (UID: \"a1ebb197-56b8-4043-be23-682f002391b5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-02665b-c768b5f9b-dpvxb" Oct 30 00:15:24.701479 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.701444 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9pc9\" (UniqueName: \"kubernetes.io/projected/a1ebb197-56b8-4043-be23-682f002391b5-kube-api-access-t9pc9\") pod \"payload-processing-e2e-worker-w4-02665b-c768b5f9b-dpvxb\" (UID: \"a1ebb197-56b8-4043-be23-682f002391b5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-02665b-c768b5f9b-dpvxb" Oct 30 00:15:24.701642 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.701591 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ebb197-56b8-4043-be23-682f002391b5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-02665b-c768b5f9b-dpvxb\" (UID: \"a1ebb197-56b8-4043-be23-682f002391b5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-02665b-c768b5f9b-dpvxb" Oct 30 00:15:24.702496 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.702468 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ebb197-56b8-4043-be23-682f002391b5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-02665b-c768b5f9b-dpvxb\" (UID: \"a1ebb197-56b8-4043-be23-682f002391b5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-02665b-c768b5f9b-dpvxb" Oct 30 00:15:24.801017 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.800861 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9pc9\" (UniqueName: \"kubernetes.io/projected/a1ebb197-56b8-4043-be23-682f002391b5-kube-api-access-t9pc9\") pod \"payload-processing-e2e-worker-w4-02665b-c768b5f9b-dpvxb\" (UID: \"a1ebb197-56b8-4043-be23-682f002391b5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-02665b-c768b5f9b-dpvxb" Oct 30 00:15:24.851480 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.851449 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-02665b-c768b5f9b-dpvxb" Oct 30 00:15:24.929686 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:24.929648 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b2c3136-aeca-4974-bc6c-0086f1f2fcda" path="/var/lib/kubelet/pods/7b2c3136-aeca-4974-bc6c-0086f1f2fcda/volumes" Oct 30 00:15:25.175110 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:25.175071 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-02665b-c768b5f9b-dpvxb"] Oct 30 00:15:25.177558 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:15:25.177525 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ebb197_56b8_4043_be23_682f002391b5.slice/crio-a14e77c6030501c2675538a41bc53e6641bbcc1f0e56ab617d506b8c6cffaafd WatchSource:0}: Error finding container a14e77c6030501c2675538a41bc53e6641bbcc1f0e56ab617d506b8c6cffaafd: Status 404 returned error can't find the container with id a14e77c6030501c2675538a41bc53e6641bbcc1f0e56ab617d506b8c6cffaafd Oct 30 00:15:26.111370 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:26.111332 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-02665b-c768b5f9b-dpvxb" event={"ID":"a1ebb197-56b8-4043-be23-682f002391b5","Type":"ContainerStarted","Data":"5ae1cf3f2009cca2ebdd16a3ff4d148b0fde0b6a4fdccf05a1a41870fda9b028"} Oct 30 00:15:26.111806 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:26.111393 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-02665b-c768b5f9b-dpvxb" event={"ID":"a1ebb197-56b8-4043-be23-682f002391b5","Type":"ContainerStarted","Data":"a14e77c6030501c2675538a41bc53e6641bbcc1f0e56ab617d506b8c6cffaafd"} Oct 30 00:15:26.111806 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:26.111425 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-02665b-c768b5f9b-dpvxb" Oct 30 00:15:26.140097 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:26.140062 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-322d20-865f5cc95-vkvd2"] Oct 30 00:15:26.140538 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:26.140519 5001 cpu_manager.go:414] "containerMap: removing container" podUID="75ca90e5-c780-4851-92e0-f3619116c5f3" containerName="cleanup" Oct 30 00:15:26.140622 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:26.140540 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75ca90e5-c780-4851-92e0-f3619116c5f3" containerName="cleanup" Oct 30 00:15:26.140698 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:26.140685 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="75ca90e5-c780-4851-92e0-f3619116c5f3" containerName="cleanup" Oct 30 00:15:26.147979 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:26.147959 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-322d20-865f5cc95-vkvd2" Oct 30 00:15:26.148091 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:26.148007 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-02665b-c768b5f9b-dpvxb" podStartSLOduration=2.147991319 podStartE2EDuration="2.147991319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:15:26.147186254 +0000 UTC m=+641.352687436" watchObservedRunningTime="2026-10-30 00:15:26.147991319 +0000 UTC m=+641.353492508" Oct 30 00:15:26.161048 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:26.161024 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-322d20-dockercfg-qpqzv\"" Oct 30 00:15:26.161158 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:26.161117 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-322d20\"" Oct 30 00:15:26.164473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:26.164453 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-322d20-865f5cc95-vkvd2"] Oct 30 00:15:26.220512 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:26.220474 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/339eea71-a503-460d-b45e-94561c780aca-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-322d20-865f5cc95-vkvd2\" (UID: \"339eea71-a503-460d-b45e-94561c780aca\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-322d20-865f5cc95-vkvd2" Oct 30 00:15:26.220677 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:26.220522 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4k25\" (UniqueName: \"kubernetes.io/projected/339eea71-a503-460d-b45e-94561c780aca-kube-api-access-n4k25\") pod \"payload-pre-processing-e2e-worker-w2-322d20-865f5cc95-vkvd2\" (UID: \"339eea71-a503-460d-b45e-94561c780aca\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-322d20-865f5cc95-vkvd2" Oct 30 00:15:26.331951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:26.321638 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/339eea71-a503-460d-b45e-94561c780aca-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-322d20-865f5cc95-vkvd2\" (UID: \"339eea71-a503-460d-b45e-94561c780aca\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-322d20-865f5cc95-vkvd2" Oct 30 00:15:26.331951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:26.321688 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4k25\" (UniqueName: \"kubernetes.io/projected/339eea71-a503-460d-b45e-94561c780aca-kube-api-access-n4k25\") pod \"payload-pre-processing-e2e-worker-w2-322d20-865f5cc95-vkvd2\" (UID: \"339eea71-a503-460d-b45e-94561c780aca\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-322d20-865f5cc95-vkvd2" Oct 30 00:15:26.331951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:26.322988 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/339eea71-a503-460d-b45e-94561c780aca-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-322d20-865f5cc95-vkvd2\" (UID: \"339eea71-a503-460d-b45e-94561c780aca\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-322d20-865f5cc95-vkvd2" Oct 30 00:15:26.353621 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:26.353590 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4k25\" (UniqueName: \"kubernetes.io/projected/339eea71-a503-460d-b45e-94561c780aca-kube-api-access-n4k25\") pod \"payload-pre-processing-e2e-worker-w2-322d20-865f5cc95-vkvd2\" (UID: \"339eea71-a503-460d-b45e-94561c780aca\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-322d20-865f5cc95-vkvd2" Oct 30 00:15:26.462020 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:26.461928 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-322d20-865f5cc95-vkvd2" Oct 30 00:15:26.776440 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:26.776414 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-322d20-865f5cc95-vkvd2"] Oct 30 00:15:26.778896 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:15:26.778868 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod339eea71_a503_460d_b45e_94561c780aca.slice/crio-69673fca2fbdbb52d86a6d8dd4d8fd07a7557a733cd269040a22f3d3796942d5 WatchSource:0}: Error finding container 69673fca2fbdbb52d86a6d8dd4d8fd07a7557a733cd269040a22f3d3796942d5: Status 404 returned error can't find the container with id 69673fca2fbdbb52d86a6d8dd4d8fd07a7557a733cd269040a22f3d3796942d5 Oct 30 00:15:27.119057 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:27.118948 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-322d20-865f5cc95-vkvd2" event={"ID":"339eea71-a503-460d-b45e-94561c780aca","Type":"ContainerStarted","Data":"93bdcd509b78b5fe9230325d72f45e172efbeded35afbcc622e155077c609fc3"} Oct 30 00:15:27.119057 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:27.119016 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-322d20-865f5cc95-vkvd2" event={"ID":"339eea71-a503-460d-b45e-94561c780aca","Type":"ContainerStarted","Data":"69673fca2fbdbb52d86a6d8dd4d8fd07a7557a733cd269040a22f3d3796942d5"} Oct 30 00:15:27.119506 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:27.119149 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-322d20-865f5cc95-vkvd2" Oct 30 00:15:27.169342 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:27.169293 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-322d20-865f5cc95-vkvd2" podStartSLOduration=1.169281107 podStartE2EDuration="1.169281107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:15:27.167047437 +0000 UTC m=+642.372548618" watchObservedRunningTime="2026-10-30 00:15:27.169281107 +0000 UTC m=+642.374782288" Oct 30 00:15:30.575844 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:30.575807 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-564c8f-57dbdd8c55-q6gz2"] Oct 30 00:15:30.580306 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:30.580285 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-564c8f-57dbdd8c55-q6gz2" Oct 30 00:15:30.599769 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:30.599736 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-564c8f-57dbdd8c55-q6gz2"] Oct 30 00:15:30.605326 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:30.605303 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-564c8f-dockercfg-4r28w\"" Oct 30 00:15:30.648866 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:30.648831 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-564c8f\"" Oct 30 00:15:30.679393 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:30.679360 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2077ddd9-c805-453e-8611-fc8b90b3b3c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-564c8f-57dbdd8c55-q6gz2\" (UID: \"2077ddd9-c805-453e-8611-fc8b90b3b3c1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-564c8f-57dbdd8c55-q6gz2" Oct 30 00:15:30.679582 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:30.679404 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks9x7\" (UniqueName: \"kubernetes.io/projected/2077ddd9-c805-453e-8611-fc8b90b3b3c1-kube-api-access-ks9x7\") pod \"payload-pre-processing-e2e-worker-w3-564c8f-57dbdd8c55-q6gz2\" (UID: \"2077ddd9-c805-453e-8611-fc8b90b3b3c1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-564c8f-57dbdd8c55-q6gz2" Oct 30 00:15:30.780600 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:30.780566 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2077ddd9-c805-453e-8611-fc8b90b3b3c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-564c8f-57dbdd8c55-q6gz2\" (UID: \"2077ddd9-c805-453e-8611-fc8b90b3b3c1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-564c8f-57dbdd8c55-q6gz2" Oct 30 00:15:30.780600 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:30.780603 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks9x7\" (UniqueName: \"kubernetes.io/projected/2077ddd9-c805-453e-8611-fc8b90b3b3c1-kube-api-access-ks9x7\") pod \"payload-pre-processing-e2e-worker-w3-564c8f-57dbdd8c55-q6gz2\" (UID: \"2077ddd9-c805-453e-8611-fc8b90b3b3c1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-564c8f-57dbdd8c55-q6gz2" Oct 30 00:15:30.781357 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:30.781330 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2077ddd9-c805-453e-8611-fc8b90b3b3c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-564c8f-57dbdd8c55-q6gz2\" (UID: \"2077ddd9-c805-453e-8611-fc8b90b3b3c1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-564c8f-57dbdd8c55-q6gz2" Oct 30 00:15:30.803741 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:30.803719 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks9x7\" (UniqueName: \"kubernetes.io/projected/2077ddd9-c805-453e-8611-fc8b90b3b3c1-kube-api-access-ks9x7\") pod \"payload-pre-processing-e2e-worker-w3-564c8f-57dbdd8c55-q6gz2\" (UID: \"2077ddd9-c805-453e-8611-fc8b90b3b3c1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-564c8f-57dbdd8c55-q6gz2" Oct 30 00:15:30.899589 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:30.899517 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-564c8f-57dbdd8c55-q6gz2" Oct 30 00:15:31.195071 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:31.194999 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-564c8f-57dbdd8c55-q6gz2"] Oct 30 00:15:31.198408 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:15:31.198384 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2077ddd9_c805_453e_8611_fc8b90b3b3c1.slice/crio-e3b5c8be47bcabd837b7a4a187bd2bf5b3222059e24d4cb62a60963019c76ace WatchSource:0}: Error finding container e3b5c8be47bcabd837b7a4a187bd2bf5b3222059e24d4cb62a60963019c76ace: Status 404 returned error can't find the container with id e3b5c8be47bcabd837b7a4a187bd2bf5b3222059e24d4cb62a60963019c76ace Oct 30 00:15:32.150625 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:32.150589 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-564c8f-57dbdd8c55-q6gz2" event={"ID":"2077ddd9-c805-453e-8611-fc8b90b3b3c1","Type":"ContainerStarted","Data":"96a3d3e90bb2f7db6d7bd8518d9bd7080b09a4f9f963d7681f36980417f8d102"} Oct 30 00:15:32.150990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:32.150684 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-564c8f-57dbdd8c55-q6gz2" event={"ID":"2077ddd9-c805-453e-8611-fc8b90b3b3c1","Type":"ContainerStarted","Data":"e3b5c8be47bcabd837b7a4a187bd2bf5b3222059e24d4cb62a60963019c76ace"} Oct 30 00:15:32.150990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:32.150707 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-564c8f-57dbdd8c55-q6gz2" Oct 30 00:15:32.186376 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:32.186334 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-564c8f-57dbdd8c55-q6gz2" podStartSLOduration=2.186320971 podStartE2EDuration="2.186320971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:15:32.184657018 +0000 UTC m=+647.390158182" watchObservedRunningTime="2026-10-30 00:15:32.186320971 +0000 UTC m=+647.391822151" Oct 30 00:15:37.122580 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:37.122552 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-02665b-c768b5f9b-dpvxb" Oct 30 00:15:38.128817 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:38.128785 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-322d20-865f5cc95-vkvd2" Oct 30 00:15:43.160083 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:43.160050 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-564c8f-57dbdd8c55-q6gz2" Oct 30 00:15:47.322843 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:15:47.322806 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:15:52.752868 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.752831 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq"] Oct 30 00:15:52.758549 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.758526 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:15:52.782003 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.781948 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-7a7ac6\"/\"openshift-service-ca.crt\"" Oct 30 00:15:52.782173 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.782079 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-7a7ac6\"/\"premium-w1-7a7ac6-kserve-self-signed-certs\"" Oct 30 00:15:52.782173 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.782146 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-7a7ac6\"/\"default-dockercfg-hvhpt\"" Oct 30 00:15:52.782326 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.782262 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-7a7ac6\"/\"kube-root-ca.crt\"" Oct 30 00:15:52.785126 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.785096 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq"] Oct 30 00:15:52.825002 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.824972 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cc8f4cb-hkwxz"] Oct 30 00:15:52.828694 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.828678 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc8f4cb-hkwxz" Oct 30 00:15:52.848366 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.848333 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cc8f4cb-hkwxz"] Oct 30 00:15:52.854852 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.854829 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dde1d5db-cea4-4020-b843-2321a1e346c4-tmp-dir\") pod \"premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq\" (UID: \"dde1d5db-cea4-4020-b843-2321a1e346c4\") " pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:15:52.855032 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.854863 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dde1d5db-cea4-4020-b843-2321a1e346c4-dshm\") pod \"premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq\" (UID: \"dde1d5db-cea4-4020-b843-2321a1e346c4\") " pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:15:52.855032 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.854899 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dde1d5db-cea4-4020-b843-2321a1e346c4-tls-certs\") pod \"premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq\" (UID: \"dde1d5db-cea4-4020-b843-2321a1e346c4\") " pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:15:52.855156 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.855059 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ps2t\" (UniqueName: \"kubernetes.io/projected/dde1d5db-cea4-4020-b843-2321a1e346c4-kube-api-access-2ps2t\") pod \"premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq\" (UID: \"dde1d5db-cea4-4020-b843-2321a1e346c4\") " pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:15:52.855156 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.855101 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dde1d5db-cea4-4020-b843-2321a1e346c4-model-cache\") pod \"premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq\" (UID: \"dde1d5db-cea4-4020-b843-2321a1e346c4\") " pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:15:52.855156 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.855132 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dde1d5db-cea4-4020-b843-2321a1e346c4-home\") pod \"premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq\" (UID: \"dde1d5db-cea4-4020-b843-2321a1e346c4\") " pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:15:52.871285 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.871258 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:15:52.871434 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.871325 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:15:52.871533 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.871520 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:15:52.871591 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.871543 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:15:52.972425 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.969826 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ps2t\" (UniqueName: \"kubernetes.io/projected/dde1d5db-cea4-4020-b843-2321a1e346c4-kube-api-access-2ps2t\") pod \"premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq\" (UID: \"dde1d5db-cea4-4020-b843-2321a1e346c4\") " pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:15:52.972425 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.969894 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dde1d5db-cea4-4020-b843-2321a1e346c4-model-cache\") pod \"premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq\" (UID: \"dde1d5db-cea4-4020-b843-2321a1e346c4\") " pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:15:52.972425 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.969945 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dde1d5db-cea4-4020-b843-2321a1e346c4-home\") pod \"premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq\" (UID: \"dde1d5db-cea4-4020-b843-2321a1e346c4\") " pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:15:52.972425 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.969998 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dde1d5db-cea4-4020-b843-2321a1e346c4-tmp-dir\") pod \"premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq\" (UID: \"dde1d5db-cea4-4020-b843-2321a1e346c4\") " pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:15:52.972425 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.970026 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvc6t\" (UniqueName: \"kubernetes.io/projected/b9de9913-86fb-4748-b043-8c0323f4e1c6-kube-api-access-cvc6t\") pod \"maas-api-revoke-sub-7cc8f4cb-hkwxz\" (UID: \"b9de9913-86fb-4748-b043-8c0323f4e1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc8f4cb-hkwxz" Oct 30 00:15:52.972425 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.970060 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dde1d5db-cea4-4020-b843-2321a1e346c4-dshm\") pod \"premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq\" (UID: \"dde1d5db-cea4-4020-b843-2321a1e346c4\") " pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:15:52.972425 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.970084 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9de9913-86fb-4748-b043-8c0323f4e1c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cc8f4cb-hkwxz\" (UID: \"b9de9913-86fb-4748-b043-8c0323f4e1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc8f4cb-hkwxz" Oct 30 00:15:52.972425 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.970120 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dde1d5db-cea4-4020-b843-2321a1e346c4-tls-certs\") pod \"premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq\" (UID: \"dde1d5db-cea4-4020-b843-2321a1e346c4\") " pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:15:52.972425 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.970159 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9de9913-86fb-4748-b043-8c0323f4e1c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cc8f4cb-hkwxz\" (UID: \"b9de9913-86fb-4748-b043-8c0323f4e1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc8f4cb-hkwxz" Oct 30 00:15:52.972425 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.971042 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dde1d5db-cea4-4020-b843-2321a1e346c4-model-cache\") pod \"premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq\" (UID: \"dde1d5db-cea4-4020-b843-2321a1e346c4\") " pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:15:52.972425 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.971403 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dde1d5db-cea4-4020-b843-2321a1e346c4-home\") pod \"premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq\" (UID: \"dde1d5db-cea4-4020-b843-2321a1e346c4\") " pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:15:52.972425 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.971682 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dde1d5db-cea4-4020-b843-2321a1e346c4-tmp-dir\") pod \"premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq\" (UID: \"dde1d5db-cea4-4020-b843-2321a1e346c4\") " pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:15:52.988508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.988419 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dde1d5db-cea4-4020-b843-2321a1e346c4-dshm\") pod \"premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq\" (UID: \"dde1d5db-cea4-4020-b843-2321a1e346c4\") " pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:15:52.999561 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:52.999476 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ps2t\" (UniqueName: \"kubernetes.io/projected/dde1d5db-cea4-4020-b843-2321a1e346c4-kube-api-access-2ps2t\") pod \"premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq\" (UID: \"dde1d5db-cea4-4020-b843-2321a1e346c4\") " pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:15:53.006572 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:53.006400 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dde1d5db-cea4-4020-b843-2321a1e346c4-tls-certs\") pod \"premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq\" (UID: \"dde1d5db-cea4-4020-b843-2321a1e346c4\") " pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:15:53.071172 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:53.071137 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvc6t\" (UniqueName: \"kubernetes.io/projected/b9de9913-86fb-4748-b043-8c0323f4e1c6-kube-api-access-cvc6t\") pod \"maas-api-revoke-sub-7cc8f4cb-hkwxz\" (UID: \"b9de9913-86fb-4748-b043-8c0323f4e1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc8f4cb-hkwxz" Oct 30 00:15:53.071350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:53.071180 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9de9913-86fb-4748-b043-8c0323f4e1c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cc8f4cb-hkwxz\" (UID: \"b9de9913-86fb-4748-b043-8c0323f4e1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc8f4cb-hkwxz" Oct 30 00:15:53.071350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:53.071211 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9de9913-86fb-4748-b043-8c0323f4e1c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cc8f4cb-hkwxz\" (UID: \"b9de9913-86fb-4748-b043-8c0323f4e1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc8f4cb-hkwxz" Oct 30 00:15:53.071875 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:53.071854 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9de9913-86fb-4748-b043-8c0323f4e1c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cc8f4cb-hkwxz\" (UID: \"b9de9913-86fb-4748-b043-8c0323f4e1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc8f4cb-hkwxz" Oct 30 00:15:53.073505 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:53.073483 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9de9913-86fb-4748-b043-8c0323f4e1c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cc8f4cb-hkwxz\" (UID: \"b9de9913-86fb-4748-b043-8c0323f4e1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc8f4cb-hkwxz" Oct 30 00:15:53.074453 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:53.074430 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:15:53.092285 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:53.092262 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvc6t\" (UniqueName: \"kubernetes.io/projected/b9de9913-86fb-4748-b043-8c0323f4e1c6-kube-api-access-cvc6t\") pod \"maas-api-revoke-sub-7cc8f4cb-hkwxz\" (UID: \"b9de9913-86fb-4748-b043-8c0323f4e1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc8f4cb-hkwxz" Oct 30 00:15:53.141950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:53.141819 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc8f4cb-hkwxz" Oct 30 00:15:53.380786 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:53.380735 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq"] Oct 30 00:15:53.387775 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:15:53.387716 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddde1d5db_cea4_4020_b843_2321a1e346c4.slice/crio-324e3a5f3b25ba7249d7d09063fe90ec98fdc4c856128548efed31849a0d5c8f WatchSource:0}: Error finding container 324e3a5f3b25ba7249d7d09063fe90ec98fdc4c856128548efed31849a0d5c8f: Status 404 returned error can't find the container with id 324e3a5f3b25ba7249d7d09063fe90ec98fdc4c856128548efed31849a0d5c8f Oct 30 00:15:53.449855 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:53.449829 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cc8f4cb-hkwxz"] Oct 30 00:15:53.453210 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:15:53.453187 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9de9913_86fb_4748_b043_8c0323f4e1c6.slice/crio-f75b3361d7e4e7f922dbf4c658a7b958b111d83ec2650f25a666370d2e4c99f4 WatchSource:0}: Error finding container f75b3361d7e4e7f922dbf4c658a7b958b111d83ec2650f25a666370d2e4c99f4: Status 404 returned error can't find the container with id f75b3361d7e4e7f922dbf4c658a7b958b111d83ec2650f25a666370d2e4c99f4 Oct 30 00:15:54.289470 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:54.289434 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc8f4cb-hkwxz" event={"ID":"b9de9913-86fb-4748-b043-8c0323f4e1c6","Type":"ContainerStarted","Data":"c3ead80eafae4db22c8ee1ae97dc33ebe657e5556fe8b389679325f01885cb1a"} Oct 30 00:15:54.289906 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:54.289524 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc8f4cb-hkwxz" event={"ID":"b9de9913-86fb-4748-b043-8c0323f4e1c6","Type":"ContainerStarted","Data":"f75b3361d7e4e7f922dbf4c658a7b958b111d83ec2650f25a666370d2e4c99f4"} Oct 30 00:15:54.291333 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:54.291301 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" event={"ID":"dde1d5db-cea4-4020-b843-2321a1e346c4","Type":"ContainerStarted","Data":"ce73dcc9235be1427e6ac49d5fc1632c73882204fc02d2d8b1dc74b770657a4e"} Oct 30 00:15:54.291454 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:54.291339 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" event={"ID":"dde1d5db-cea4-4020-b843-2321a1e346c4","Type":"ContainerStarted","Data":"324e3a5f3b25ba7249d7d09063fe90ec98fdc4c856128548efed31849a0d5c8f"} Oct 30 00:15:54.328237 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:54.328182 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc8f4cb-hkwxz" podStartSLOduration=2.3281664539999998 podStartE2EDuration="2.328166454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:15:54.326370877 +0000 UTC m=+669.531872059" watchObservedRunningTime="2026-10-30 00:15:54.328166454 +0000 UTC m=+669.533667637" Oct 30 00:15:54.364769 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:54.364723 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podStartSLOduration=2.364707374 podStartE2EDuration="2.364707374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:15:54.362584683 +0000 UTC m=+669.568085863" watchObservedRunningTime="2026-10-30 00:15:54.364707374 +0000 UTC m=+669.570208557" Oct 30 00:15:55.075024 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:55.074989 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:15:55.299381 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:55.299346 5001 generic.go:356] "Generic (PLEG): container finished" podID="b9de9913-86fb-4748-b043-8c0323f4e1c6" containerID="c3ead80eafae4db22c8ee1ae97dc33ebe657e5556fe8b389679325f01885cb1a" exitCode=0 Oct 30 00:15:55.299772 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:55.299430 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc8f4cb-hkwxz" event={"ID":"b9de9913-86fb-4748-b043-8c0323f4e1c6","Type":"ContainerDied","Data":"c3ead80eafae4db22c8ee1ae97dc33ebe657e5556fe8b389679325f01885cb1a"} Oct 30 00:15:56.446593 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:56.446567 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc8f4cb-hkwxz" Oct 30 00:15:56.608318 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:56.608238 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9de9913-86fb-4748-b043-8c0323f4e1c6-kube-api-access-cvc6t\" (UniqueName: \"kubernetes.io/projected/b9de9913-86fb-4748-b043-8c0323f4e1c6-kube-api-access-cvc6t\") pod \"b9de9913-86fb-4748-b043-8c0323f4e1c6\" (UID: \"b9de9913-86fb-4748-b043-8c0323f4e1c6\") " Oct 30 00:15:56.608318 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:56.608288 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9de9913-86fb-4748-b043-8c0323f4e1c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9de9913-86fb-4748-b043-8c0323f4e1c6-maas-api-xks-ca\") pod \"b9de9913-86fb-4748-b043-8c0323f4e1c6\" (UID: \"b9de9913-86fb-4748-b043-8c0323f4e1c6\") " Oct 30 00:15:56.608511 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:56.608385 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9de9913-86fb-4748-b043-8c0323f4e1c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9de9913-86fb-4748-b043-8c0323f4e1c6-maas-api-openshift-ca\") pod \"b9de9913-86fb-4748-b043-8c0323f4e1c6\" (UID: \"b9de9913-86fb-4748-b043-8c0323f4e1c6\") " Oct 30 00:15:56.608759 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:56.608726 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9de9913-86fb-4748-b043-8c0323f4e1c6-maas-api-openshift-ca" pod "b9de9913-86fb-4748-b043-8c0323f4e1c6" (UID: "b9de9913-86fb-4748-b043-8c0323f4e1c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:15:56.610369 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:56.610347 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9de9913-86fb-4748-b043-8c0323f4e1c6-maas-api-xks-ca" pod "b9de9913-86fb-4748-b043-8c0323f4e1c6" (UID: "b9de9913-86fb-4748-b043-8c0323f4e1c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:15:56.610452 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:56.610364 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9de9913-86fb-4748-b043-8c0323f4e1c6-kube-api-access-cvc6t" pod "b9de9913-86fb-4748-b043-8c0323f4e1c6" (UID: "b9de9913-86fb-4748-b043-8c0323f4e1c6"). InnerVolumeSpecName "kube-api-access-cvc6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:15:56.709878 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:56.709842 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9de9913-86fb-4748-b043-8c0323f4e1c6-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:15:56.709878 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:56.709872 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvc6t\" (UniqueName: \"kubernetes.io/projected/b9de9913-86fb-4748-b043-8c0323f4e1c6-kube-api-access-cvc6t\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:15:56.709878 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:56.709886 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9de9913-86fb-4748-b043-8c0323f4e1c6-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:15:57.312460 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:57.312432 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc8f4cb-hkwxz" Oct 30 00:15:57.312626 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:57.312471 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc8f4cb-hkwxz" event={"ID":"b9de9913-86fb-4748-b043-8c0323f4e1c6","Type":"ContainerDied","Data":"f75b3361d7e4e7f922dbf4c658a7b958b111d83ec2650f25a666370d2e4c99f4"} Oct 30 00:15:57.312626 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:57.312518 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f75b3361d7e4e7f922dbf4c658a7b958b111d83ec2650f25a666370d2e4c99f4" Oct 30 00:15:58.608645 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.608613 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl"] Oct 30 00:15:58.609191 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.609098 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b9de9913-86fb-4748-b043-8c0323f4e1c6" containerName="revoke-keys" Oct 30 00:15:58.609191 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.609112 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9de9913-86fb-4748-b043-8c0323f4e1c6" containerName="revoke-keys" Oct 30 00:15:58.609307 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.609217 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9de9913-86fb-4748-b043-8c0323f4e1c6" containerName="revoke-keys" Oct 30 00:15:58.613461 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.613440 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:15:58.637168 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.637141 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-6c45bb-openshift-default-dockercfg-d5ghz\"" Oct 30 00:15:58.650096 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.650067 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl"] Oct 30 00:15:58.731741 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.731697 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:15:58.731946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.731782 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:15:58.732008 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.731940 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:15:58.732008 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.731969 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:15:58.732008 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.732000 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcxnd\" (UniqueName: \"kubernetes.io/projected/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-kube-api-access-xcxnd\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:15:58.732121 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.732035 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:15:58.732121 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.732064 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:15:58.732121 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.732097 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:15:58.732225 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.732182 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:15:58.833604 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.833568 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:15:58.833800 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.833624 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:15:58.833800 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.833656 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:15:58.833800 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.833691 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:15:58.833800 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.833707 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:15:58.833800 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.833728 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcxnd\" (UniqueName: \"kubernetes.io/projected/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-kube-api-access-xcxnd\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:15:58.833800 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.833757 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:15:58.833800 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.833785 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:15:58.834230 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.833820 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:15:58.834230 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.834193 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:15:58.834452 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.834427 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:15:58.834637 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.834488 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:15:58.834637 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.834500 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:15:58.834840 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.834822 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:15:58.836046 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.836026 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:15:58.836463 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.836445 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:15:58.854308 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.854282 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:15:58.854589 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.854569 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcxnd\" (UniqueName: \"kubernetes.io/projected/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-kube-api-access-xcxnd\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:15:58.941605 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:58.941522 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:15:59.266566 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:59.266528 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl"] Oct 30 00:15:59.270809 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:15:59.270783 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd5b4b9_3358_4549_80ba_5b84960fd3c0.slice/crio-0feb8030ebec2f98b0429e42ae77a290dc5c6da9971c139247cbb8479cf6f6c5 WatchSource:0}: Error finding container 0feb8030ebec2f98b0429e42ae77a290dc5c6da9971c139247cbb8479cf6f6c5: Status 404 returned error can't find the container with id 0feb8030ebec2f98b0429e42ae77a290dc5c6da9971c139247cbb8479cf6f6c5 Oct 30 00:15:59.273239 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:59.273199 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:15:59.273308 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:59.273290 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:15:59.273344 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:59.273334 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:15:59.326992 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:15:59.326961 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" event={"ID":"bfd5b4b9-3358-4549-80ba-5b84960fd3c0","Type":"ContainerStarted","Data":"0feb8030ebec2f98b0429e42ae77a290dc5c6da9971c139247cbb8479cf6f6c5"} Oct 30 00:16:00.335492 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:00.335450 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" event={"ID":"bfd5b4b9-3358-4549-80ba-5b84960fd3c0","Type":"ContainerStarted","Data":"1e2e0a73b6b9055d8b01356a1d7027e4571dd7d1d26adcb0f4cc8fb5ecf352a0"} Oct 30 00:16:00.399183 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:00.399129 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" podStartSLOduration=2.399114858 podStartE2EDuration="2.399114858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:16:00.387335363 +0000 UTC m=+675.592836555" watchObservedRunningTime="2026-10-30 00:16:00.399114858 +0000 UTC m=+675.604616039" Oct 30 00:16:00.944976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:00.944503 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:16:00.944976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:00.944537 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75"] Oct 30 00:16:00.944976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:00.944558 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" podUID="bfd5b4b9-3358-4549-80ba-5b84960fd3c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 00:16:00.948993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:00.948973 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:16:00.949124 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:00.949086 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:16:00.963745 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:00.963722 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75"] Oct 30 00:16:00.967520 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:00.967498 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-7505bd-openshift-default-dockercfg-98hdd\"" Oct 30 00:16:01.060599 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.060558 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c415517-5484-4905-b9df-8510d1da9ddb-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:16:01.060801 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.060640 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvllh\" (UniqueName: \"kubernetes.io/projected/4c415517-5484-4905-b9df-8510d1da9ddb-kube-api-access-gvllh\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:16:01.060861 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.060824 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:16:01.060940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.060876 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:16:01.060999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.060939 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:16:01.060999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.060977 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:16:01.061099 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.061000 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:16:01.061099 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.061087 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c415517-5484-4905-b9df-8510d1da9ddb-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:16:01.061200 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.061143 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c415517-5484-4905-b9df-8510d1da9ddb-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:16:01.162643 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.162608 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:16:01.162822 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.162662 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:16:01.162822 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.162699 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c415517-5484-4905-b9df-8510d1da9ddb-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:16:01.162822 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.162773 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c415517-5484-4905-b9df-8510d1da9ddb-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:16:01.163092 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.162827 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c415517-5484-4905-b9df-8510d1da9ddb-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:16:01.163092 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.162873 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvllh\" (UniqueName: \"kubernetes.io/projected/4c415517-5484-4905-b9df-8510d1da9ddb-kube-api-access-gvllh\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:16:01.163092 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.162943 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:16:01.163092 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.162980 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:16:01.163092 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.163010 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:16:01.163401 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.163105 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:16:01.163401 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.163150 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:16:01.163401 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.163371 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:16:01.163521 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.163413 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:16:01.163635 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.163612 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c415517-5484-4905-b9df-8510d1da9ddb-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:16:01.165224 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.165195 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:16:01.165942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.165923 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c415517-5484-4905-b9df-8510d1da9ddb-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:16:01.185358 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.185333 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c415517-5484-4905-b9df-8510d1da9ddb-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:16:01.187778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.187758 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvllh\" (UniqueName: \"kubernetes.io/projected/4c415517-5484-4905-b9df-8510d1da9ddb-kube-api-access-gvllh\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:16:01.267663 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.267631 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:16:01.636670 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.636577 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75"] Oct 30 00:16:01.645605 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:16:01.645524 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c415517_5484_4905_b9df_8510d1da9ddb.slice/crio-25abc2bb48d9409bdb2fe9cf34760f424568e8ffda0e20ac9f83e58d37c2d180 WatchSource:0}: Error finding container 25abc2bb48d9409bdb2fe9cf34760f424568e8ffda0e20ac9f83e58d37c2d180: Status 404 returned error can't find the container with id 25abc2bb48d9409bdb2fe9cf34760f424568e8ffda0e20ac9f83e58d37c2d180 Oct 30 00:16:01.648339 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.648304 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:16:01.648466 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.648389 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:16:01.648466 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.648422 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:16:01.944381 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.944293 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:16:01.944827 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:01.944789 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" podUID="bfd5b4b9-3358-4549-80ba-5b84960fd3c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 00:16:02.365780 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:02.365581 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" event={"ID":"4c415517-5484-4905-b9df-8510d1da9ddb","Type":"ContainerStarted","Data":"d65dbb115be2d9cf05d380376a09bec489fc95b4bdcb9ff3a78dba621359c3ba"} Oct 30 00:16:02.365780 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:02.365687 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" event={"ID":"4c415517-5484-4905-b9df-8510d1da9ddb","Type":"ContainerStarted","Data":"25abc2bb48d9409bdb2fe9cf34760f424568e8ffda0e20ac9f83e58d37c2d180"} Oct 30 00:16:02.452938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:02.452004 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" podStartSLOduration=2.451981742 podStartE2EDuration="2.451981742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:16:02.424522473 +0000 UTC m=+677.630023681" watchObservedRunningTime="2026-10-30 00:16:02.451981742 +0000 UTC m=+677.657482925" Oct 30 00:16:02.957939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:02.945406 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:16:02.957939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:02.945483 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" podUID="bfd5b4b9-3358-4549-80ba-5b84960fd3c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 00:16:03.075461 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:03.075385 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:16:03.089660 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:03.089632 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:16:03.268933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:03.268883 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:16:03.270477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:03.270451 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 00:16:03.270544 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:03.270497 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" podUID="4c415517-5484-4905-b9df-8510d1da9ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 00:16:03.384039 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:03.384004 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:16:03.943342 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:03.943304 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:16:03.943651 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:03.943623 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" podUID="bfd5b4b9-3358-4549-80ba-5b84960fd3c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 00:16:04.268705 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:04.268669 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 00:16:04.269200 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:04.268726 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" podUID="4c415517-5484-4905-b9df-8510d1da9ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 00:16:04.944934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:04.944273 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:16:04.944934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:04.944330 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" podUID="bfd5b4b9-3358-4549-80ba-5b84960fd3c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 00:16:05.274962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:05.269294 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 00:16:05.274962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:05.269394 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" podUID="4c415517-5484-4905-b9df-8510d1da9ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 00:16:05.942411 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:05.942375 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:16:05.942728 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:05.942697 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" podUID="bfd5b4b9-3358-4549-80ba-5b84960fd3c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 00:16:06.268385 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:06.268357 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 00:16:06.268547 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:06.268421 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" podUID="4c415517-5484-4905-b9df-8510d1da9ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 00:16:06.947030 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:06.947000 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:16:06.947426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:06.947258 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:16:06.948026 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:06.948010 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:16:07.268189 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:07.268155 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 00:16:07.268389 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:07.268222 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" podUID="4c415517-5484-4905-b9df-8510d1da9ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 00:16:07.271805 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:07.271779 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a51b204e-4g5zc"] Oct 30 00:16:07.275821 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:07.275806 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b204e-4g5zc" Oct 30 00:16:07.290141 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:07.290119 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:16:07.290301 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:07.290125 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:16:07.290403 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:07.290177 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:16:07.290572 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:07.290274 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:16:07.294010 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:07.293989 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a51b204e-4g5zc"] Oct 30 00:16:07.296813 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:07.296791 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d57fa402-3023-47ed-b4ff-2b90737d422c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a51b204e-4g5zc\" (UID: \"d57fa402-3023-47ed-b4ff-2b90737d422c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b204e-4g5zc" Oct 30 00:16:07.296905 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:07.296830 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d57fa402-3023-47ed-b4ff-2b90737d422c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a51b204e-4g5zc\" (UID: \"d57fa402-3023-47ed-b4ff-2b90737d422c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b204e-4g5zc" Oct 30 00:16:07.296905 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:07.296853 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khspc\" (UniqueName: \"kubernetes.io/projected/d57fa402-3023-47ed-b4ff-2b90737d422c-kube-api-access-khspc\") pod \"maas-api-revoke-sub-a51b204e-4g5zc\" (UID: \"d57fa402-3023-47ed-b4ff-2b90737d422c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b204e-4g5zc" Oct 30 00:16:07.397653 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:07.397622 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d57fa402-3023-47ed-b4ff-2b90737d422c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a51b204e-4g5zc\" (UID: \"d57fa402-3023-47ed-b4ff-2b90737d422c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b204e-4g5zc" Oct 30 00:16:07.397816 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:07.397667 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d57fa402-3023-47ed-b4ff-2b90737d422c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a51b204e-4g5zc\" (UID: \"d57fa402-3023-47ed-b4ff-2b90737d422c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b204e-4g5zc" Oct 30 00:16:07.397816 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:07.397708 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khspc\" (UniqueName: \"kubernetes.io/projected/d57fa402-3023-47ed-b4ff-2b90737d422c-kube-api-access-khspc\") pod \"maas-api-revoke-sub-a51b204e-4g5zc\" (UID: \"d57fa402-3023-47ed-b4ff-2b90737d422c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b204e-4g5zc" Oct 30 00:16:07.398362 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:07.398339 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d57fa402-3023-47ed-b4ff-2b90737d422c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a51b204e-4g5zc\" (UID: \"d57fa402-3023-47ed-b4ff-2b90737d422c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b204e-4g5zc" Oct 30 00:16:07.400014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:07.399996 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d57fa402-3023-47ed-b4ff-2b90737d422c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a51b204e-4g5zc\" (UID: \"d57fa402-3023-47ed-b4ff-2b90737d422c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b204e-4g5zc" Oct 30 00:16:07.449933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:07.449763 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khspc\" (UniqueName: \"kubernetes.io/projected/d57fa402-3023-47ed-b4ff-2b90737d422c-kube-api-access-khspc\") pod \"maas-api-revoke-sub-a51b204e-4g5zc\" (UID: \"d57fa402-3023-47ed-b4ff-2b90737d422c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b204e-4g5zc" Oct 30 00:16:07.589496 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:07.589415 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b204e-4g5zc" Oct 30 00:16:07.905943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:07.905843 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a51b204e-4g5zc"] Oct 30 00:16:07.909788 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:16:07.909749 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd57fa402_3023_47ed_b4ff_2b90737d422c.slice/crio-9913a5bd1a60eac3a2295dcf294e28929396517e21a8fef12ac1701eb26381f6 WatchSource:0}: Error finding container 9913a5bd1a60eac3a2295dcf294e28929396517e21a8fef12ac1701eb26381f6: Status 404 returned error can't find the container with id 9913a5bd1a60eac3a2295dcf294e28929396517e21a8fef12ac1701eb26381f6 Oct 30 00:16:08.443945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:08.441555 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b204e-4g5zc" event={"ID":"d57fa402-3023-47ed-b4ff-2b90737d422c","Type":"ContainerStarted","Data":"2a7437a334123b1c3b1c87b92aed6e64d3d0d6877952d29e1d42c623344804d9"} Oct 30 00:16:08.443945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:08.441617 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b204e-4g5zc" event={"ID":"d57fa402-3023-47ed-b4ff-2b90737d422c","Type":"ContainerStarted","Data":"9913a5bd1a60eac3a2295dcf294e28929396517e21a8fef12ac1701eb26381f6"} Oct 30 00:16:08.481260 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:08.481196 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b204e-4g5zc" podStartSLOduration=1.481176843 podStartE2EDuration="1.481176843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:16:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:16:08.476699666 +0000 UTC m=+683.682200849" watchObservedRunningTime="2026-10-30 00:16:08.481176843 +0000 UTC m=+683.686678037" Oct 30 00:16:08.729449 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:08.729370 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5ad099d-2698c"] Oct 30 00:16:08.733452 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:08.733428 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ad099d-2698c" Oct 30 00:16:08.752335 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:08.752267 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5ad099d-2698c"] Oct 30 00:16:08.814334 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:08.814299 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1b6516-758d-4a1d-a89b-0198c9d3ad46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5ad099d-2698c\" (UID: \"af1b6516-758d-4a1d-a89b-0198c9d3ad46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ad099d-2698c" Oct 30 00:16:08.814334 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:08.814342 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1b6516-758d-4a1d-a89b-0198c9d3ad46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5ad099d-2698c\" (UID: \"af1b6516-758d-4a1d-a89b-0198c9d3ad46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ad099d-2698c" Oct 30 00:16:08.814615 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:08.814451 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcqph\" (UniqueName: \"kubernetes.io/projected/af1b6516-758d-4a1d-a89b-0198c9d3ad46-kube-api-access-pcqph\") pod \"maas-api-revoke-sub-c5ad099d-2698c\" (UID: \"af1b6516-758d-4a1d-a89b-0198c9d3ad46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ad099d-2698c" Oct 30 00:16:08.959955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:08.958894 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1b6516-758d-4a1d-a89b-0198c9d3ad46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5ad099d-2698c\" (UID: \"af1b6516-758d-4a1d-a89b-0198c9d3ad46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ad099d-2698c" Oct 30 00:16:08.959955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:08.959165 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1b6516-758d-4a1d-a89b-0198c9d3ad46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5ad099d-2698c\" (UID: \"af1b6516-758d-4a1d-a89b-0198c9d3ad46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ad099d-2698c" Oct 30 00:16:08.959955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:08.959598 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcqph\" (UniqueName: \"kubernetes.io/projected/af1b6516-758d-4a1d-a89b-0198c9d3ad46-kube-api-access-pcqph\") pod \"maas-api-revoke-sub-c5ad099d-2698c\" (UID: \"af1b6516-758d-4a1d-a89b-0198c9d3ad46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ad099d-2698c" Oct 30 00:16:08.968931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:08.963322 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1b6516-758d-4a1d-a89b-0198c9d3ad46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5ad099d-2698c\" (UID: \"af1b6516-758d-4a1d-a89b-0198c9d3ad46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ad099d-2698c" Oct 30 00:16:08.974394 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:08.974323 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1b6516-758d-4a1d-a89b-0198c9d3ad46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5ad099d-2698c\" (UID: \"af1b6516-758d-4a1d-a89b-0198c9d3ad46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ad099d-2698c" Oct 30 00:16:09.029960 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:09.019372 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcqph\" (UniqueName: \"kubernetes.io/projected/af1b6516-758d-4a1d-a89b-0198c9d3ad46-kube-api-access-pcqph\") pod \"maas-api-revoke-sub-c5ad099d-2698c\" (UID: \"af1b6516-758d-4a1d-a89b-0198c9d3ad46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ad099d-2698c" Oct 30 00:16:09.053359 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:09.053129 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ad099d-2698c" Oct 30 00:16:09.280313 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:09.280211 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:16:09.280313 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:09.280279 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" podUID="4c415517-5484-4905-b9df-8510d1da9ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:16:09.282249 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:09.282219 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:16:09.282935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:09.282902 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:16:09.284625 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:09.284591 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:16:09.421287 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:09.421255 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5ad099d-2698c"] Oct 30 00:16:09.422774 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:16:09.422744 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf1b6516_758d_4a1d_a89b_0198c9d3ad46.slice/crio-0d7b3e7c0f95d0325f6e09aa977a44ffda78051c7e6815a72c0a54ddac6f5646 WatchSource:0}: Error finding container 0d7b3e7c0f95d0325f6e09aa977a44ffda78051c7e6815a72c0a54ddac6f5646: Status 404 returned error can't find the container with id 0d7b3e7c0f95d0325f6e09aa977a44ffda78051c7e6815a72c0a54ddac6f5646 Oct 30 00:16:09.445468 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:09.445442 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ad099d-2698c" event={"ID":"af1b6516-758d-4a1d-a89b-0198c9d3ad46","Type":"ContainerStarted","Data":"0d7b3e7c0f95d0325f6e09aa977a44ffda78051c7e6815a72c0a54ddac6f5646"} Oct 30 00:16:09.447328 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:09.447302 5001 generic.go:356] "Generic (PLEG): container finished" podID="d57fa402-3023-47ed-b4ff-2b90737d422c" containerID="2a7437a334123b1c3b1c87b92aed6e64d3d0d6877952d29e1d42c623344804d9" exitCode=0 Oct 30 00:16:09.447418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:09.447378 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b204e-4g5zc" event={"ID":"d57fa402-3023-47ed-b4ff-2b90737d422c","Type":"ContainerDied","Data":"2a7437a334123b1c3b1c87b92aed6e64d3d0d6877952d29e1d42c623344804d9"} Oct 30 00:16:10.456098 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:10.456063 5001 generic.go:356] "Generic (PLEG): container finished" podID="af1b6516-758d-4a1d-a89b-0198c9d3ad46" containerID="53246042e69856a9043b1257163f17639ad44a4d09ac4a00319709b4dee277cd" exitCode=0 Oct 30 00:16:10.456491 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:10.456149 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ad099d-2698c" event={"ID":"af1b6516-758d-4a1d-a89b-0198c9d3ad46","Type":"ContainerDied","Data":"53246042e69856a9043b1257163f17639ad44a4d09ac4a00319709b4dee277cd"} Oct 30 00:16:10.608446 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:10.608422 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b204e-4g5zc" Oct 30 00:16:10.680550 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:10.680504 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d57fa402-3023-47ed-b4ff-2b90737d422c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d57fa402-3023-47ed-b4ff-2b90737d422c-maas-api-openshift-ca\") pod \"d57fa402-3023-47ed-b4ff-2b90737d422c\" (UID: \"d57fa402-3023-47ed-b4ff-2b90737d422c\") " Oct 30 00:16:10.680746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:10.680662 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d57fa402-3023-47ed-b4ff-2b90737d422c-kube-api-access-khspc\" (UniqueName: \"kubernetes.io/projected/d57fa402-3023-47ed-b4ff-2b90737d422c-kube-api-access-khspc\") pod \"d57fa402-3023-47ed-b4ff-2b90737d422c\" (UID: \"d57fa402-3023-47ed-b4ff-2b90737d422c\") " Oct 30 00:16:10.680810 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:10.680771 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d57fa402-3023-47ed-b4ff-2b90737d422c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d57fa402-3023-47ed-b4ff-2b90737d422c-maas-api-xks-ca\") pod \"d57fa402-3023-47ed-b4ff-2b90737d422c\" (UID: \"d57fa402-3023-47ed-b4ff-2b90737d422c\") " Oct 30 00:16:10.683374 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:10.683346 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d57fa402-3023-47ed-b4ff-2b90737d422c-maas-api-openshift-ca" pod "d57fa402-3023-47ed-b4ff-2b90737d422c" (UID: "d57fa402-3023-47ed-b4ff-2b90737d422c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:16:10.684059 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:10.684034 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d57fa402-3023-47ed-b4ff-2b90737d422c-maas-api-xks-ca" pod "d57fa402-3023-47ed-b4ff-2b90737d422c" (UID: "d57fa402-3023-47ed-b4ff-2b90737d422c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:16:10.685194 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:10.685169 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d57fa402-3023-47ed-b4ff-2b90737d422c-kube-api-access-khspc" pod "d57fa402-3023-47ed-b4ff-2b90737d422c" (UID: "d57fa402-3023-47ed-b4ff-2b90737d422c"). InnerVolumeSpecName "kube-api-access-khspc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:16:10.782190 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:10.782108 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d57fa402-3023-47ed-b4ff-2b90737d422c-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:16:10.782190 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:10.782141 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khspc\" (UniqueName: \"kubernetes.io/projected/d57fa402-3023-47ed-b4ff-2b90737d422c-kube-api-access-khspc\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:16:10.782190 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:10.782151 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d57fa402-3023-47ed-b4ff-2b90737d422c-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:16:11.464239 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:11.464201 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b204e-4g5zc" event={"ID":"d57fa402-3023-47ed-b4ff-2b90737d422c","Type":"ContainerDied","Data":"9913a5bd1a60eac3a2295dcf294e28929396517e21a8fef12ac1701eb26381f6"} Oct 30 00:16:11.464239 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:11.464223 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b204e-4g5zc" Oct 30 00:16:11.464893 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:11.464258 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9913a5bd1a60eac3a2295dcf294e28929396517e21a8fef12ac1701eb26381f6" Oct 30 00:16:11.602747 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:11.602724 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ad099d-2698c" Oct 30 00:16:11.692102 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:11.692071 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af1b6516-758d-4a1d-a89b-0198c9d3ad46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1b6516-758d-4a1d-a89b-0198c9d3ad46-maas-api-openshift-ca\") pod \"af1b6516-758d-4a1d-a89b-0198c9d3ad46\" (UID: \"af1b6516-758d-4a1d-a89b-0198c9d3ad46\") " Oct 30 00:16:11.692289 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:11.692109 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af1b6516-758d-4a1d-a89b-0198c9d3ad46-kube-api-access-pcqph\" (UniqueName: \"kubernetes.io/projected/af1b6516-758d-4a1d-a89b-0198c9d3ad46-kube-api-access-pcqph\") pod \"af1b6516-758d-4a1d-a89b-0198c9d3ad46\" (UID: \"af1b6516-758d-4a1d-a89b-0198c9d3ad46\") " Oct 30 00:16:11.692289 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:11.692132 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af1b6516-758d-4a1d-a89b-0198c9d3ad46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1b6516-758d-4a1d-a89b-0198c9d3ad46-maas-api-xks-ca\") pod \"af1b6516-758d-4a1d-a89b-0198c9d3ad46\" (UID: \"af1b6516-758d-4a1d-a89b-0198c9d3ad46\") " Oct 30 00:16:11.692571 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:11.692529 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af1b6516-758d-4a1d-a89b-0198c9d3ad46-maas-api-openshift-ca" pod "af1b6516-758d-4a1d-a89b-0198c9d3ad46" (UID: "af1b6516-758d-4a1d-a89b-0198c9d3ad46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:16:11.694421 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:11.694391 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af1b6516-758d-4a1d-a89b-0198c9d3ad46-maas-api-xks-ca" pod "af1b6516-758d-4a1d-a89b-0198c9d3ad46" (UID: "af1b6516-758d-4a1d-a89b-0198c9d3ad46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:16:11.694421 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:11.694409 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af1b6516-758d-4a1d-a89b-0198c9d3ad46-kube-api-access-pcqph" pod "af1b6516-758d-4a1d-a89b-0198c9d3ad46" (UID: "af1b6516-758d-4a1d-a89b-0198c9d3ad46"). InnerVolumeSpecName "kube-api-access-pcqph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:16:11.793907 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:11.793870 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1b6516-758d-4a1d-a89b-0198c9d3ad46-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:16:11.793907 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:11.793902 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcqph\" (UniqueName: \"kubernetes.io/projected/af1b6516-758d-4a1d-a89b-0198c9d3ad46-kube-api-access-pcqph\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:16:11.794137 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:11.793943 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1b6516-758d-4a1d-a89b-0198c9d3ad46-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:16:12.471968 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:12.471942 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ad099d-2698c" Oct 30 00:16:12.472353 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:12.471971 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ad099d-2698c" event={"ID":"af1b6516-758d-4a1d-a89b-0198c9d3ad46","Type":"ContainerDied","Data":"0d7b3e7c0f95d0325f6e09aa977a44ffda78051c7e6815a72c0a54ddac6f5646"} Oct 30 00:16:12.472353 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:12.472019 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d7b3e7c0f95d0325f6e09aa977a44ffda78051c7e6815a72c0a54ddac6f5646" Oct 30 00:16:17.321791 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:16:17.321748 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:16:23.031423 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:23.031390 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6ad69a-w2bh7"] Oct 30 00:16:23.032054 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:23.032034 5001 cpu_manager.go:414] "containerMap: removing container" podUID="af1b6516-758d-4a1d-a89b-0198c9d3ad46" containerName="revoke-keys" Oct 30 00:16:23.032154 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:23.032056 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af1b6516-758d-4a1d-a89b-0198c9d3ad46" containerName="revoke-keys" Oct 30 00:16:23.032154 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:23.032107 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d57fa402-3023-47ed-b4ff-2b90737d422c" containerName="revoke-keys" Oct 30 00:16:23.032154 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:23.032116 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d57fa402-3023-47ed-b4ff-2b90737d422c" containerName="revoke-keys" Oct 30 00:16:23.032313 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:23.032254 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d57fa402-3023-47ed-b4ff-2b90737d422c" containerName="revoke-keys" Oct 30 00:16:23.032313 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:23.032275 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="af1b6516-758d-4a1d-a89b-0198c9d3ad46" containerName="revoke-keys" Oct 30 00:16:23.035697 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:23.035674 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6ad69a-w2bh7" Oct 30 00:16:23.052860 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:23.052832 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6ad69a-w2bh7"] Oct 30 00:16:23.059007 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:23.058980 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:16:23.059160 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:23.059137 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:16:23.059160 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:23.059089 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:16:23.059340 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:23.059231 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:16:23.122370 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:23.122335 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c8366d-ccbd-4047-8172-38a7a9d29909-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d6ad69a-w2bh7\" (UID: \"e9c8366d-ccbd-4047-8172-38a7a9d29909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6ad69a-w2bh7" Oct 30 00:16:23.122527 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:23.122400 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c75sx\" (UniqueName: \"kubernetes.io/projected/e9c8366d-ccbd-4047-8172-38a7a9d29909-kube-api-access-c75sx\") pod \"maas-api-revoke-sub-9d6ad69a-w2bh7\" (UID: \"e9c8366d-ccbd-4047-8172-38a7a9d29909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6ad69a-w2bh7" Oct 30 00:16:23.122527 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:23.122489 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c8366d-ccbd-4047-8172-38a7a9d29909-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d6ad69a-w2bh7\" (UID: \"e9c8366d-ccbd-4047-8172-38a7a9d29909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6ad69a-w2bh7" Oct 30 00:16:23.223797 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:23.223761 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c8366d-ccbd-4047-8172-38a7a9d29909-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d6ad69a-w2bh7\" (UID: \"e9c8366d-ccbd-4047-8172-38a7a9d29909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6ad69a-w2bh7" Oct 30 00:16:23.223977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:23.223821 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c75sx\" (UniqueName: \"kubernetes.io/projected/e9c8366d-ccbd-4047-8172-38a7a9d29909-kube-api-access-c75sx\") pod \"maas-api-revoke-sub-9d6ad69a-w2bh7\" (UID: \"e9c8366d-ccbd-4047-8172-38a7a9d29909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6ad69a-w2bh7" Oct 30 00:16:23.223977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:23.223843 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c8366d-ccbd-4047-8172-38a7a9d29909-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d6ad69a-w2bh7\" (UID: \"e9c8366d-ccbd-4047-8172-38a7a9d29909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6ad69a-w2bh7" Oct 30 00:16:23.224419 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:23.224396 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c8366d-ccbd-4047-8172-38a7a9d29909-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d6ad69a-w2bh7\" (UID: \"e9c8366d-ccbd-4047-8172-38a7a9d29909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6ad69a-w2bh7" Oct 30 00:16:23.226225 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:23.226202 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c8366d-ccbd-4047-8172-38a7a9d29909-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d6ad69a-w2bh7\" (UID: \"e9c8366d-ccbd-4047-8172-38a7a9d29909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6ad69a-w2bh7" Oct 30 00:16:23.244589 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:23.244564 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c75sx\" (UniqueName: \"kubernetes.io/projected/e9c8366d-ccbd-4047-8172-38a7a9d29909-kube-api-access-c75sx\") pod \"maas-api-revoke-sub-9d6ad69a-w2bh7\" (UID: \"e9c8366d-ccbd-4047-8172-38a7a9d29909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6ad69a-w2bh7" Oct 30 00:16:23.349133 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:23.349060 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6ad69a-w2bh7" Oct 30 00:16:23.648854 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:23.648769 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6ad69a-w2bh7"] Oct 30 00:16:23.652832 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:16:23.652801 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c8366d_ccbd_4047_8172_38a7a9d29909.slice/crio-4c2dd800240f95fddcb0a5af433ad498662090779b45602a872b4672811bdf13 WatchSource:0}: Error finding container 4c2dd800240f95fddcb0a5af433ad498662090779b45602a872b4672811bdf13: Status 404 returned error can't find the container with id 4c2dd800240f95fddcb0a5af433ad498662090779b45602a872b4672811bdf13 Oct 30 00:16:24.563268 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:24.563232 5001 generic.go:356] "Generic (PLEG): container finished" podID="e9c8366d-ccbd-4047-8172-38a7a9d29909" containerID="70f766da62595ccde16900c36c0bf36c6e30c5b17046f62c8498d9cd275cae99" exitCode=0 Oct 30 00:16:24.563676 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:24.563268 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6ad69a-w2bh7" event={"ID":"e9c8366d-ccbd-4047-8172-38a7a9d29909","Type":"ContainerDied","Data":"70f766da62595ccde16900c36c0bf36c6e30c5b17046f62c8498d9cd275cae99"} Oct 30 00:16:24.563676 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:24.563372 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6ad69a-w2bh7" event={"ID":"e9c8366d-ccbd-4047-8172-38a7a9d29909","Type":"ContainerStarted","Data":"4c2dd800240f95fddcb0a5af433ad498662090779b45602a872b4672811bdf13"} Oct 30 00:16:24.813537 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:24.813501 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-493311b8-62hdl"] Oct 30 00:16:24.817329 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:24.817311 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493311b8-62hdl" Oct 30 00:16:24.835172 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:24.835143 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-493311b8-62hdl"] Oct 30 00:16:24.946827 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:24.946790 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a72a2dca-19ca-4a54-a217-936ef3a2e613-maas-api-xks-ca\") pod \"maas-api-revoke-sub-493311b8-62hdl\" (UID: \"a72a2dca-19ca-4a54-a217-936ef3a2e613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493311b8-62hdl" Oct 30 00:16:24.947000 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:24.946851 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a72a2dca-19ca-4a54-a217-936ef3a2e613-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-493311b8-62hdl\" (UID: \"a72a2dca-19ca-4a54-a217-936ef3a2e613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493311b8-62hdl" Oct 30 00:16:24.947000 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:24.946954 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjjb5\" (UniqueName: \"kubernetes.io/projected/a72a2dca-19ca-4a54-a217-936ef3a2e613-kube-api-access-tjjb5\") pod \"maas-api-revoke-sub-493311b8-62hdl\" (UID: \"a72a2dca-19ca-4a54-a217-936ef3a2e613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493311b8-62hdl" Oct 30 00:16:25.047694 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:25.047660 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a72a2dca-19ca-4a54-a217-936ef3a2e613-maas-api-xks-ca\") pod \"maas-api-revoke-sub-493311b8-62hdl\" (UID: \"a72a2dca-19ca-4a54-a217-936ef3a2e613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493311b8-62hdl" Oct 30 00:16:25.047857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:25.047752 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a72a2dca-19ca-4a54-a217-936ef3a2e613-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-493311b8-62hdl\" (UID: \"a72a2dca-19ca-4a54-a217-936ef3a2e613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493311b8-62hdl" Oct 30 00:16:25.047857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:25.047790 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjjb5\" (UniqueName: \"kubernetes.io/projected/a72a2dca-19ca-4a54-a217-936ef3a2e613-kube-api-access-tjjb5\") pod \"maas-api-revoke-sub-493311b8-62hdl\" (UID: \"a72a2dca-19ca-4a54-a217-936ef3a2e613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493311b8-62hdl" Oct 30 00:16:25.048535 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:25.048511 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a72a2dca-19ca-4a54-a217-936ef3a2e613-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-493311b8-62hdl\" (UID: \"a72a2dca-19ca-4a54-a217-936ef3a2e613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493311b8-62hdl" Oct 30 00:16:25.050153 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:25.050132 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a72a2dca-19ca-4a54-a217-936ef3a2e613-maas-api-xks-ca\") pod \"maas-api-revoke-sub-493311b8-62hdl\" (UID: \"a72a2dca-19ca-4a54-a217-936ef3a2e613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493311b8-62hdl" Oct 30 00:16:25.068350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:25.068289 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjjb5\" (UniqueName: \"kubernetes.io/projected/a72a2dca-19ca-4a54-a217-936ef3a2e613-kube-api-access-tjjb5\") pod \"maas-api-revoke-sub-493311b8-62hdl\" (UID: \"a72a2dca-19ca-4a54-a217-936ef3a2e613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493311b8-62hdl" Oct 30 00:16:25.130646 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:25.130612 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493311b8-62hdl" Oct 30 00:16:25.427513 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:25.427484 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-493311b8-62hdl"] Oct 30 00:16:25.429364 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:16:25.429330 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda72a2dca_19ca_4a54_a217_936ef3a2e613.slice/crio-17ca40c302c8749ed4890218f509718b98c0cef525711076fa78f9ff9edfd538 WatchSource:0}: Error finding container 17ca40c302c8749ed4890218f509718b98c0cef525711076fa78f9ff9edfd538: Status 404 returned error can't find the container with id 17ca40c302c8749ed4890218f509718b98c0cef525711076fa78f9ff9edfd538 Oct 30 00:16:25.571533 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:25.571498 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493311b8-62hdl" event={"ID":"a72a2dca-19ca-4a54-a217-936ef3a2e613","Type":"ContainerStarted","Data":"17ca40c302c8749ed4890218f509718b98c0cef525711076fa78f9ff9edfd538"} Oct 30 00:16:25.728123 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:25.728100 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6ad69a-w2bh7" Oct 30 00:16:25.856516 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:25.856480 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9c8366d-ccbd-4047-8172-38a7a9d29909-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c8366d-ccbd-4047-8172-38a7a9d29909-maas-api-openshift-ca\") pod \"e9c8366d-ccbd-4047-8172-38a7a9d29909\" (UID: \"e9c8366d-ccbd-4047-8172-38a7a9d29909\") " Oct 30 00:16:25.856708 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:25.856576 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9c8366d-ccbd-4047-8172-38a7a9d29909-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c8366d-ccbd-4047-8172-38a7a9d29909-maas-api-xks-ca\") pod \"e9c8366d-ccbd-4047-8172-38a7a9d29909\" (UID: \"e9c8366d-ccbd-4047-8172-38a7a9d29909\") " Oct 30 00:16:25.856708 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:25.856601 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c8366d-ccbd-4047-8172-38a7a9d29909-kube-api-access-c75sx\" (UniqueName: \"kubernetes.io/projected/e9c8366d-ccbd-4047-8172-38a7a9d29909-kube-api-access-c75sx\") pod \"e9c8366d-ccbd-4047-8172-38a7a9d29909\" (UID: \"e9c8366d-ccbd-4047-8172-38a7a9d29909\") " Oct 30 00:16:25.856871 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:25.856832 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9c8366d-ccbd-4047-8172-38a7a9d29909-maas-api-openshift-ca" pod "e9c8366d-ccbd-4047-8172-38a7a9d29909" (UID: "e9c8366d-ccbd-4047-8172-38a7a9d29909"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:16:25.857001 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:25.856961 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c8366d-ccbd-4047-8172-38a7a9d29909-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:16:25.858699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:25.858675 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9c8366d-ccbd-4047-8172-38a7a9d29909-maas-api-xks-ca" pod "e9c8366d-ccbd-4047-8172-38a7a9d29909" (UID: "e9c8366d-ccbd-4047-8172-38a7a9d29909"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:16:25.858781 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:25.858734 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c8366d-ccbd-4047-8172-38a7a9d29909-kube-api-access-c75sx" pod "e9c8366d-ccbd-4047-8172-38a7a9d29909" (UID: "e9c8366d-ccbd-4047-8172-38a7a9d29909"). InnerVolumeSpecName "kube-api-access-c75sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:16:25.957670 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:25.957597 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c8366d-ccbd-4047-8172-38a7a9d29909-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:16:25.957670 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:25.957623 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c75sx\" (UniqueName: \"kubernetes.io/projected/e9c8366d-ccbd-4047-8172-38a7a9d29909-kube-api-access-c75sx\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:16:26.579560 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:26.579531 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6ad69a-w2bh7" Oct 30 00:16:26.579560 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:26.579552 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6ad69a-w2bh7" event={"ID":"e9c8366d-ccbd-4047-8172-38a7a9d29909","Type":"ContainerDied","Data":"4c2dd800240f95fddcb0a5af433ad498662090779b45602a872b4672811bdf13"} Oct 30 00:16:26.580057 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:26.579596 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c2dd800240f95fddcb0a5af433ad498662090779b45602a872b4672811bdf13" Oct 30 00:16:26.585386 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:26.585357 5001 generic.go:356] "Generic (PLEG): container finished" podID="a72a2dca-19ca-4a54-a217-936ef3a2e613" containerID="f625604a8d269b50f9fc112ed41688f3616dec972d54a3f4a850973a666e7e68" exitCode=0 Oct 30 00:16:26.585545 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:26.585432 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493311b8-62hdl" event={"ID":"a72a2dca-19ca-4a54-a217-936ef3a2e613","Type":"ContainerDied","Data":"f625604a8d269b50f9fc112ed41688f3616dec972d54a3f4a850973a666e7e68"} Oct 30 00:16:27.820961 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:27.820936 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493311b8-62hdl" Oct 30 00:16:27.877188 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:27.877106 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a72a2dca-19ca-4a54-a217-936ef3a2e613-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a72a2dca-19ca-4a54-a217-936ef3a2e613-maas-api-xks-ca\") pod \"a72a2dca-19ca-4a54-a217-936ef3a2e613\" (UID: \"a72a2dca-19ca-4a54-a217-936ef3a2e613\") " Oct 30 00:16:27.877354 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:27.877234 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a72a2dca-19ca-4a54-a217-936ef3a2e613-kube-api-access-tjjb5\" (UniqueName: \"kubernetes.io/projected/a72a2dca-19ca-4a54-a217-936ef3a2e613-kube-api-access-tjjb5\") pod \"a72a2dca-19ca-4a54-a217-936ef3a2e613\" (UID: \"a72a2dca-19ca-4a54-a217-936ef3a2e613\") " Oct 30 00:16:27.879272 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:27.879234 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a72a2dca-19ca-4a54-a217-936ef3a2e613-maas-api-xks-ca" pod "a72a2dca-19ca-4a54-a217-936ef3a2e613" (UID: "a72a2dca-19ca-4a54-a217-936ef3a2e613"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:16:27.879406 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:27.879299 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a72a2dca-19ca-4a54-a217-936ef3a2e613-kube-api-access-tjjb5" pod "a72a2dca-19ca-4a54-a217-936ef3a2e613" (UID: "a72a2dca-19ca-4a54-a217-936ef3a2e613"). InnerVolumeSpecName "kube-api-access-tjjb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:16:27.978799 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:27.978015 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a72a2dca-19ca-4a54-a217-936ef3a2e613-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a72a2dca-19ca-4a54-a217-936ef3a2e613-maas-api-openshift-ca\") pod \"a72a2dca-19ca-4a54-a217-936ef3a2e613\" (UID: \"a72a2dca-19ca-4a54-a217-936ef3a2e613\") " Oct 30 00:16:27.978799 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:27.978444 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a72a2dca-19ca-4a54-a217-936ef3a2e613-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:16:27.978799 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:27.978473 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjjb5\" (UniqueName: \"kubernetes.io/projected/a72a2dca-19ca-4a54-a217-936ef3a2e613-kube-api-access-tjjb5\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:16:27.979773 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:27.979735 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a72a2dca-19ca-4a54-a217-936ef3a2e613-maas-api-openshift-ca" pod "a72a2dca-19ca-4a54-a217-936ef3a2e613" (UID: "a72a2dca-19ca-4a54-a217-936ef3a2e613"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:16:28.080071 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:28.080030 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a72a2dca-19ca-4a54-a217-936ef3a2e613-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:16:28.600392 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:28.600363 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493311b8-62hdl" Oct 30 00:16:28.600593 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:28.600400 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493311b8-62hdl" event={"ID":"a72a2dca-19ca-4a54-a217-936ef3a2e613","Type":"ContainerDied","Data":"17ca40c302c8749ed4890218f509718b98c0cef525711076fa78f9ff9edfd538"} Oct 30 00:16:28.600593 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:28.600454 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17ca40c302c8749ed4890218f509718b98c0cef525711076fa78f9ff9edfd538" Oct 30 00:16:35.130838 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:35.130805 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eebf0d77-h7c9s"] Oct 30 00:16:35.131319 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:35.131284 5001 cpu_manager.go:414] "containerMap: removing container" podUID="a72a2dca-19ca-4a54-a217-936ef3a2e613" containerName="revoke-keys" Oct 30 00:16:35.131319 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:35.131296 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a72a2dca-19ca-4a54-a217-936ef3a2e613" containerName="revoke-keys" Oct 30 00:16:35.131319 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:35.131314 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e9c8366d-ccbd-4047-8172-38a7a9d29909" containerName="revoke-keys" Oct 30 00:16:35.131319 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:35.131319 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9c8366d-ccbd-4047-8172-38a7a9d29909" containerName="revoke-keys" Oct 30 00:16:35.131451 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:35.131413 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a72a2dca-19ca-4a54-a217-936ef3a2e613" containerName="revoke-keys" Oct 30 00:16:35.131451 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:35.131423 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9c8366d-ccbd-4047-8172-38a7a9d29909" containerName="revoke-keys" Oct 30 00:16:35.134998 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:35.134978 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebf0d77-h7c9s" Oct 30 00:16:35.146448 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:35.146422 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f91c3a1-351a-4939-86c8-f6864f38b6b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eebf0d77-h7c9s\" (UID: \"7f91c3a1-351a-4939-86c8-f6864f38b6b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebf0d77-h7c9s" Oct 30 00:16:35.146552 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:35.146487 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f91c3a1-351a-4939-86c8-f6864f38b6b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eebf0d77-h7c9s\" (UID: \"7f91c3a1-351a-4939-86c8-f6864f38b6b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebf0d77-h7c9s" Oct 30 00:16:35.146624 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:35.146609 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xz6g\" (UniqueName: \"kubernetes.io/projected/7f91c3a1-351a-4939-86c8-f6864f38b6b3-kube-api-access-6xz6g\") pod \"maas-api-revoke-sub-eebf0d77-h7c9s\" (UID: \"7f91c3a1-351a-4939-86c8-f6864f38b6b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebf0d77-h7c9s" Oct 30 00:16:35.150004 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:35.149981 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:16:35.159743 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:35.159713 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:16:35.159967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:35.159945 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:16:35.160058 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:35.159967 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:16:35.164980 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:35.164957 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eebf0d77-h7c9s"] Oct 30 00:16:35.247932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:35.247884 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f91c3a1-351a-4939-86c8-f6864f38b6b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eebf0d77-h7c9s\" (UID: \"7f91c3a1-351a-4939-86c8-f6864f38b6b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebf0d77-h7c9s" Oct 30 00:16:35.248122 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:35.247960 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f91c3a1-351a-4939-86c8-f6864f38b6b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eebf0d77-h7c9s\" (UID: \"7f91c3a1-351a-4939-86c8-f6864f38b6b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebf0d77-h7c9s" Oct 30 00:16:35.248122 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:35.248097 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xz6g\" (UniqueName: \"kubernetes.io/projected/7f91c3a1-351a-4939-86c8-f6864f38b6b3-kube-api-access-6xz6g\") pod \"maas-api-revoke-sub-eebf0d77-h7c9s\" (UID: \"7f91c3a1-351a-4939-86c8-f6864f38b6b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebf0d77-h7c9s" Oct 30 00:16:35.248713 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:35.248694 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f91c3a1-351a-4939-86c8-f6864f38b6b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eebf0d77-h7c9s\" (UID: \"7f91c3a1-351a-4939-86c8-f6864f38b6b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebf0d77-h7c9s" Oct 30 00:16:35.250256 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:35.250232 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f91c3a1-351a-4939-86c8-f6864f38b6b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eebf0d77-h7c9s\" (UID: \"7f91c3a1-351a-4939-86c8-f6864f38b6b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebf0d77-h7c9s" Oct 30 00:16:35.268799 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:35.268769 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xz6g\" (UniqueName: \"kubernetes.io/projected/7f91c3a1-351a-4939-86c8-f6864f38b6b3-kube-api-access-6xz6g\") pod \"maas-api-revoke-sub-eebf0d77-h7c9s\" (UID: \"7f91c3a1-351a-4939-86c8-f6864f38b6b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebf0d77-h7c9s" Oct 30 00:16:35.450436 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:35.450349 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebf0d77-h7c9s" Oct 30 00:16:35.749390 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:35.749366 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eebf0d77-h7c9s"] Oct 30 00:16:35.751846 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:16:35.751819 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f91c3a1_351a_4939_86c8_f6864f38b6b3.slice/crio-d9eab88620f1842f1a516746caff16393098c55c77ca188b4e0ed7178c138755 WatchSource:0}: Error finding container d9eab88620f1842f1a516746caff16393098c55c77ca188b4e0ed7178c138755: Status 404 returned error can't find the container with id d9eab88620f1842f1a516746caff16393098c55c77ca188b4e0ed7178c138755 Oct 30 00:16:36.653156 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.653123 5001 generic.go:356] "Generic (PLEG): container finished" podID="7f91c3a1-351a-4939-86c8-f6864f38b6b3" containerID="b87a6e349eb5724d27cab5601555fd0c0affbc3008df7277dd054b8017bb4c60" exitCode=0 Oct 30 00:16:36.653515 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.653200 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebf0d77-h7c9s" event={"ID":"7f91c3a1-351a-4939-86c8-f6864f38b6b3","Type":"ContainerDied","Data":"b87a6e349eb5724d27cab5601555fd0c0affbc3008df7277dd054b8017bb4c60"} Oct 30 00:16:36.653515 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.653252 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebf0d77-h7c9s" event={"ID":"7f91c3a1-351a-4939-86c8-f6864f38b6b3","Type":"ContainerStarted","Data":"d9eab88620f1842f1a516746caff16393098c55c77ca188b4e0ed7178c138755"} Oct 30 00:16:36.758965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.758928 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx"] Oct 30 00:16:36.762761 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.762741 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:16:36.775246 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.775221 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-b1c718-openshift-default-dockercfg-8spcf\"" Oct 30 00:16:36.785476 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.785449 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx"] Oct 30 00:16:36.862588 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.862492 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:16:36.862588 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.862546 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:16:36.862786 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.862645 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:16:36.862786 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.862692 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:16:36.862879 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.862798 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:16:36.862879 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.862836 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2823a5-cf70-47df-8d48-430910c2023b-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:16:36.862879 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.862862 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa2823a5-cf70-47df-8d48-430910c2023b-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:16:36.863079 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.863033 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2823a5-cf70-47df-8d48-430910c2023b-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:16:36.863079 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.863064 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42fpc\" (UniqueName: \"kubernetes.io/projected/aa2823a5-cf70-47df-8d48-430910c2023b-kube-api-access-42fpc\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:16:36.964560 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.964527 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:16:36.964560 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.964568 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2823a5-cf70-47df-8d48-430910c2023b-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:16:36.964771 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.964592 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa2823a5-cf70-47df-8d48-430910c2023b-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:16:36.964771 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.964653 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2823a5-cf70-47df-8d48-430910c2023b-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:16:36.964771 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.964684 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42fpc\" (UniqueName: \"kubernetes.io/projected/aa2823a5-cf70-47df-8d48-430910c2023b-kube-api-access-42fpc\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:16:36.964771 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.964738 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:16:36.965021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.964778 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:16:36.965021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.964815 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:16:36.965021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.964885 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:16:36.965021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.965005 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:16:36.965338 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.965320 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:16:36.965404 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.965372 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:16:36.965457 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.965404 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:16:36.965615 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.965599 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2823a5-cf70-47df-8d48-430910c2023b-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:16:36.967011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.966987 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:16:36.967251 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.967233 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2823a5-cf70-47df-8d48-430910c2023b-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:16:36.985130 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.985102 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa2823a5-cf70-47df-8d48-430910c2023b-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:16:36.985413 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:36.985393 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42fpc\" (UniqueName: \"kubernetes.io/projected/aa2823a5-cf70-47df-8d48-430910c2023b-kube-api-access-42fpc\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:16:37.075861 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:37.075829 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:16:37.393749 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:37.393660 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx"] Oct 30 00:16:37.396875 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:16:37.396846 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa2823a5_cf70_47df_8d48_430910c2023b.slice/crio-6e6309b8b073bcdb40d0b3772b49289acc1edef220c03687bcdb674950ed8f6f WatchSource:0}: Error finding container 6e6309b8b073bcdb40d0b3772b49289acc1edef220c03687bcdb674950ed8f6f: Status 404 returned error can't find the container with id 6e6309b8b073bcdb40d0b3772b49289acc1edef220c03687bcdb674950ed8f6f Oct 30 00:16:37.399692 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:37.399659 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:16:37.399804 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:37.399717 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:16:37.399804 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:37.399751 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:16:37.661240 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:37.661148 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" event={"ID":"aa2823a5-cf70-47df-8d48-430910c2023b","Type":"ContainerStarted","Data":"ea2e13577e89d03389a30c125dd7d5ff47cd19b80ea8c843324e3f5624104f5c"} Oct 30 00:16:37.661240 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:37.661227 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" event={"ID":"aa2823a5-cf70-47df-8d48-430910c2023b","Type":"ContainerStarted","Data":"6e6309b8b073bcdb40d0b3772b49289acc1edef220c03687bcdb674950ed8f6f"} Oct 30 00:16:37.702645 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:37.702577 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" podStartSLOduration=1.7025588740000002 podStartE2EDuration="1.702558874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:16:37.700083579 +0000 UTC m=+712.905584786" watchObservedRunningTime="2026-10-30 00:16:37.702558874 +0000 UTC m=+712.908060056" Oct 30 00:16:37.814625 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:37.814600 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebf0d77-h7c9s" Oct 30 00:16:37.874498 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:37.874465 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f91c3a1-351a-4939-86c8-f6864f38b6b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f91c3a1-351a-4939-86c8-f6864f38b6b3-maas-api-xks-ca\") pod \"7f91c3a1-351a-4939-86c8-f6864f38b6b3\" (UID: \"7f91c3a1-351a-4939-86c8-f6864f38b6b3\") " Oct 30 00:16:37.874673 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:37.874580 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f91c3a1-351a-4939-86c8-f6864f38b6b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f91c3a1-351a-4939-86c8-f6864f38b6b3-maas-api-openshift-ca\") pod \"7f91c3a1-351a-4939-86c8-f6864f38b6b3\" (UID: \"7f91c3a1-351a-4939-86c8-f6864f38b6b3\") " Oct 30 00:16:37.874673 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:37.874628 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f91c3a1-351a-4939-86c8-f6864f38b6b3-kube-api-access-6xz6g\" (UniqueName: \"kubernetes.io/projected/7f91c3a1-351a-4939-86c8-f6864f38b6b3-kube-api-access-6xz6g\") pod \"7f91c3a1-351a-4939-86c8-f6864f38b6b3\" (UID: \"7f91c3a1-351a-4939-86c8-f6864f38b6b3\") " Oct 30 00:16:37.875157 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:37.874963 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f91c3a1-351a-4939-86c8-f6864f38b6b3-maas-api-openshift-ca" pod "7f91c3a1-351a-4939-86c8-f6864f38b6b3" (UID: "7f91c3a1-351a-4939-86c8-f6864f38b6b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:16:37.875157 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:37.875129 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f91c3a1-351a-4939-86c8-f6864f38b6b3-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:16:37.876681 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:37.876659 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f91c3a1-351a-4939-86c8-f6864f38b6b3-maas-api-xks-ca" pod "7f91c3a1-351a-4939-86c8-f6864f38b6b3" (UID: "7f91c3a1-351a-4939-86c8-f6864f38b6b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:16:37.877220 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:37.877196 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f91c3a1-351a-4939-86c8-f6864f38b6b3-kube-api-access-6xz6g" pod "7f91c3a1-351a-4939-86c8-f6864f38b6b3" (UID: "7f91c3a1-351a-4939-86c8-f6864f38b6b3"). InnerVolumeSpecName "kube-api-access-6xz6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:16:37.976746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:37.976659 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xz6g\" (UniqueName: \"kubernetes.io/projected/7f91c3a1-351a-4939-86c8-f6864f38b6b3-kube-api-access-6xz6g\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:16:37.976746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:37.976691 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f91c3a1-351a-4939-86c8-f6864f38b6b3-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:16:38.076412 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.076372 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:16:38.078027 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.077999 5001 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 00:16:38.078165 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.078055 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" podUID="aa2823a5-cf70-47df-8d48-430910c2023b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 00:16:38.300451 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.300415 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx"] Oct 30 00:16:38.300952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.300927 5001 cpu_manager.go:414] "containerMap: removing container" podUID="7f91c3a1-351a-4939-86c8-f6864f38b6b3" containerName="revoke-keys" Oct 30 00:16:38.300952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.300954 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f91c3a1-351a-4939-86c8-f6864f38b6b3" containerName="revoke-keys" Oct 30 00:16:38.301068 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.301039 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f91c3a1-351a-4939-86c8-f6864f38b6b3" containerName="revoke-keys" Oct 30 00:16:38.304583 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.304564 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:16:38.321981 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.319526 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-7a7ac6-openshift-default-dockercfg-rv5lp\"" Oct 30 00:16:38.342161 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.342128 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx"] Oct 30 00:16:38.383081 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.383053 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:16:38.383081 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.383087 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:16:38.383317 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.383109 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2qb2\" (UniqueName: \"kubernetes.io/projected/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-kube-api-access-r2qb2\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:16:38.383317 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.383164 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:16:38.383317 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.383193 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:16:38.383484 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.383332 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:16:38.383484 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.383377 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:16:38.383580 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.383495 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:16:38.383580 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.383557 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:16:38.484365 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.484334 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:16:38.484560 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.484373 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:16:38.484560 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.484397 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2qb2\" (UniqueName: \"kubernetes.io/projected/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-kube-api-access-r2qb2\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:16:38.484560 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.484415 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:16:38.484560 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.484434 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:16:38.484560 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.484484 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:16:38.484560 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.484527 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:16:38.484853 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.484596 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:16:38.484853 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.484625 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:16:38.484853 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.484825 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:16:38.485033 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.484877 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:16:38.485033 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.484976 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:16:38.485143 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.485123 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:16:38.485301 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.485278 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:16:38.486938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.486886 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:16:38.487233 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.487216 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:16:38.553935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.533107 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2qb2\" (UniqueName: \"kubernetes.io/projected/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-kube-api-access-r2qb2\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:16:38.557934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.556880 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:16:38.626169 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.626124 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:16:38.670025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.670000 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebf0d77-h7c9s" Oct 30 00:16:38.670419 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.670074 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebf0d77-h7c9s" event={"ID":"7f91c3a1-351a-4939-86c8-f6864f38b6b3","Type":"ContainerDied","Data":"d9eab88620f1842f1a516746caff16393098c55c77ca188b4e0ed7178c138755"} Oct 30 00:16:38.670419 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.670124 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9eab88620f1842f1a516746caff16393098c55c77ca188b4e0ed7178c138755" Oct 30 00:16:38.954620 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.954563 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx"] Oct 30 00:16:38.957157 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:16:38.957132 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad5882d9_90e8_400f_90d5_d4ba5d8da5fc.slice/crio-a9c41186ebf661d7a0e35ca6699bcb3657725614e1b5d2e6202dcfab6f1b733f WatchSource:0}: Error finding container a9c41186ebf661d7a0e35ca6699bcb3657725614e1b5d2e6202dcfab6f1b733f: Status 404 returned error can't find the container with id a9c41186ebf661d7a0e35ca6699bcb3657725614e1b5d2e6202dcfab6f1b733f Oct 30 00:16:38.959490 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.959455 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:16:38.959553 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.959528 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:16:38.959594 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:38.959558 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:16:39.076323 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:39.076288 5001 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 00:16:39.076505 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:39.076350 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" podUID="aa2823a5-cf70-47df-8d48-430910c2023b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 00:16:39.677552 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:39.677508 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" event={"ID":"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc","Type":"ContainerStarted","Data":"cf90f41004529a7c7f998958db88becac65c20dfc62fc7a12df95ca23f4d05fa"} Oct 30 00:16:39.677977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:39.677566 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" event={"ID":"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc","Type":"ContainerStarted","Data":"a9c41186ebf661d7a0e35ca6699bcb3657725614e1b5d2e6202dcfab6f1b733f"} Oct 30 00:16:39.743451 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:39.743390 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" podStartSLOduration=1.743367869 podStartE2EDuration="1.743367869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:16:39.728441545 +0000 UTC m=+714.933942727" watchObservedRunningTime="2026-10-30 00:16:39.743367869 +0000 UTC m=+714.948869051" Oct 30 00:16:39.943991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:39.934813 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s"] Oct 30 00:16:39.957081 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:39.948173 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:16:39.957081 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:39.953939 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s"] Oct 30 00:16:39.961242 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:39.961216 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-564c8f-openshift-default-dockercfg-f7gl6\"" Oct 30 00:16:40.002689 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.002649 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:16:40.002689 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.002692 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:16:40.002943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.002709 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx6ct\" (UniqueName: \"kubernetes.io/projected/f549cc40-a1ae-43aa-825f-11d538b9fa71-kube-api-access-gx6ct\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:16:40.002943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.002785 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:16:40.003064 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.002945 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:16:40.003064 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.002978 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f549cc40-a1ae-43aa-825f-11d538b9fa71-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:16:40.003064 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.003010 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:16:40.003064 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.003032 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f549cc40-a1ae-43aa-825f-11d538b9fa71-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:16:40.003064 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.003054 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f549cc40-a1ae-43aa-825f-11d538b9fa71-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:16:40.076471 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.076433 5001 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 00:16:40.076660 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.076489 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" podUID="aa2823a5-cf70-47df-8d48-430910c2023b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 00:16:40.104246 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.104210 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:16:40.104405 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.104290 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:16:40.104405 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.104327 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f549cc40-a1ae-43aa-825f-11d538b9fa71-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:16:40.104405 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.104358 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:16:40.104405 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.104389 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f549cc40-a1ae-43aa-825f-11d538b9fa71-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:16:40.104625 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.104420 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f549cc40-a1ae-43aa-825f-11d538b9fa71-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:16:40.104625 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.104464 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:16:40.104625 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.104493 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:16:40.104625 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.104517 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx6ct\" (UniqueName: \"kubernetes.io/projected/f549cc40-a1ae-43aa-825f-11d538b9fa71-kube-api-access-gx6ct\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:16:40.104809 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.104686 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:16:40.105275 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.105248 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:16:40.105530 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.105512 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:16:40.105685 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.105637 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f549cc40-a1ae-43aa-825f-11d538b9fa71-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:16:40.105854 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.105835 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:16:40.107515 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.107491 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f549cc40-a1ae-43aa-825f-11d538b9fa71-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:16:40.107702 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.107683 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:16:40.147790 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.147751 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx6ct\" (UniqueName: \"kubernetes.io/projected/f549cc40-a1ae-43aa-825f-11d538b9fa71-kube-api-access-gx6ct\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:16:40.148468 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.148404 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f549cc40-a1ae-43aa-825f-11d538b9fa71-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:16:40.277324 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.277289 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:16:40.642993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.634749 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:16:40.642993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.638573 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 00:16:40.642993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.638629 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" podUID="ad5882d9-90e8-400f-90d5-d4ba5d8da5fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 00:16:40.744934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.744384 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s"] Oct 30 00:16:40.865942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.835691 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:16:40.865942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.835853 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:16:40.865942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:40.835969 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:16:41.080933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:41.079309 5001 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 00:16:41.080933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:41.079375 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" podUID="aa2823a5-cf70-47df-8d48-430910c2023b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 00:16:41.634345 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:41.634304 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 00:16:41.634623 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:41.634584 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" podUID="ad5882d9-90e8-400f-90d5-d4ba5d8da5fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 00:16:41.704692 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:41.704648 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" event={"ID":"f549cc40-a1ae-43aa-825f-11d538b9fa71","Type":"ContainerStarted","Data":"d93b8a9f02863d14cd884a190ee8aebf784d7e11d8b488b2342554bccacab816"} Oct 30 00:16:41.704891 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:41.704714 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" event={"ID":"f549cc40-a1ae-43aa-825f-11d538b9fa71","Type":"ContainerStarted","Data":"d3baae909e1014fd5313c7d5e01e1613d6aa5c8fefb782e73679e6c8457beee6"} Oct 30 00:16:41.764931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:41.764456 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" podStartSLOduration=2.764433747 podStartE2EDuration="2.764433747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:16:41.756399283 +0000 UTC m=+716.961900465" watchObservedRunningTime="2026-10-30 00:16:41.764433747 +0000 UTC m=+716.969934931" Oct 30 00:16:42.077324 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:42.077288 5001 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 00:16:42.077558 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:42.077536 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" podUID="aa2823a5-cf70-47df-8d48-430910c2023b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 00:16:42.279937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:42.279245 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:16:42.284083 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:42.284050 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 00:16:42.284324 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:42.284297 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" podUID="f549cc40-a1ae-43aa-825f-11d538b9fa71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 00:16:42.633104 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:42.633065 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 00:16:42.633403 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:42.633364 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" podUID="ad5882d9-90e8-400f-90d5-d4ba5d8da5fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 00:16:43.096101 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:43.096065 5001 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 00:16:43.096696 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:43.096661 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" podUID="aa2823a5-cf70-47df-8d48-430910c2023b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 00:16:43.278439 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:43.278402 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 00:16:43.278730 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:43.278700 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" podUID="f549cc40-a1ae-43aa-825f-11d538b9fa71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 00:16:43.639353 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:43.639312 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 00:16:43.639656 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:43.639626 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" podUID="ad5882d9-90e8-400f-90d5-d4ba5d8da5fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 00:16:44.076552 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:44.076514 5001 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 00:16:44.076730 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:44.076569 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" podUID="aa2823a5-cf70-47df-8d48-430910c2023b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 00:16:44.278669 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:44.278630 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 00:16:44.279070 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:44.278680 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" podUID="f549cc40-a1ae-43aa-825f-11d538b9fa71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 00:16:44.638060 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:44.638020 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 00:16:44.638346 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:44.638317 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" podUID="ad5882d9-90e8-400f-90d5-d4ba5d8da5fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 00:16:45.080265 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:45.080242 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:16:45.080756 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:45.080742 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:16:45.081363 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:45.081344 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:16:45.277843 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:45.277813 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 00:16:45.278076 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:45.278054 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" podUID="f549cc40-a1ae-43aa-825f-11d538b9fa71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 00:16:45.638936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:45.638290 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 00:16:45.638936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:45.638355 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" podUID="ad5882d9-90e8-400f-90d5-d4ba5d8da5fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 00:16:46.278693 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:46.278658 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 00:16:46.278879 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:46.278718 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" podUID="f549cc40-a1ae-43aa-825f-11d538b9fa71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 00:16:46.636846 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:46.636767 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:16:46.637206 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:46.637188 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:16:46.637797 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:46.637781 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:16:47.278370 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:47.278333 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 00:16:47.278764 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:47.278387 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" podUID="f549cc40-a1ae-43aa-825f-11d538b9fa71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 00:16:47.347417 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:16:47.347380 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:16:48.282808 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:48.282779 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:16:48.283230 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:48.283211 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:16:48.283895 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:16:48.283877 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:17:03.730571 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:03.730455 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb"] Oct 30 00:17:03.734758 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:03.734737 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:17:03.747324 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:03.747300 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-02665b-openshift-default-dockercfg-c72fk\"" Oct 30 00:17:03.761062 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:03.761041 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb"] Oct 30 00:17:03.887956 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:03.887899 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istio-podinfo\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:17:03.888119 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:03.887968 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgvdx\" (UniqueName: \"kubernetes.io/projected/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-kube-api-access-rgvdx\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:17:03.888119 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:03.888034 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-workload-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:17:03.888119 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:03.888054 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istiod-ca-cert\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:17:03.888119 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:03.888077 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istio-data\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:17:03.888275 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:03.888207 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-workload-certs\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:17:03.888275 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:03.888239 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istio-token\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:17:03.888275 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:03.888267 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-credential-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:17:03.888391 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:03.888377 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istio-envoy\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:17:03.989309 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:03.989229 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istio-token\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:17:03.989309 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:03.989278 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-credential-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:17:03.989497 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:03.989461 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istio-envoy\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:17:03.989544 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:03.989531 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istio-podinfo\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:17:03.989581 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:03.989551 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgvdx\" (UniqueName: \"kubernetes.io/projected/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-kube-api-access-rgvdx\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:17:03.989581 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:03.989577 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-workload-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:17:03.989656 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:03.989607 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istiod-ca-cert\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:17:03.989656 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:03.989643 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istio-data\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:17:03.989800 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:03.989688 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-credential-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:17:03.989800 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:03.989718 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-workload-certs\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:17:03.990016 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:03.989997 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-workload-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:17:03.990107 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:03.990011 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istio-data\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:17:03.990107 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:03.990071 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-workload-certs\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:17:03.990329 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:03.990312 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istiod-ca-cert\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:17:03.991769 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:03.991753 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istio-envoy\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:17:03.991965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:03.991946 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istio-podinfo\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:17:04.010142 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:04.010119 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istio-token\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:17:04.010251 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:04.010139 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgvdx\" (UniqueName: \"kubernetes.io/projected/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-kube-api-access-rgvdx\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:17:04.047412 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:04.047390 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:17:04.370319 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:04.370293 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb"] Oct 30 00:17:04.372694 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:17:04.372666 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode29579fa_0beb_4dea_8cde_bf5cd16afb5d.slice/crio-7e96a29e9439e0361b1688914adef84e54b33a520e973fe0884f37bda5858a69 WatchSource:0}: Error finding container 7e96a29e9439e0361b1688914adef84e54b33a520e973fe0884f37bda5858a69: Status 404 returned error can't find the container with id 7e96a29e9439e0361b1688914adef84e54b33a520e973fe0884f37bda5858a69 Oct 30 00:17:04.375173 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:04.375140 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:17:04.375282 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:04.375219 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:17:04.375282 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:04.375266 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:17:05.013947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:05.008319 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" event={"ID":"e29579fa-0beb-4dea-8cde-bf5cd16afb5d","Type":"ContainerStarted","Data":"2b9de37a96bc5d5e1d3867bfe8ed70d9be1d9ba00468ace2a905c399c76128bd"} Oct 30 00:17:05.014661 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:05.014627 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" event={"ID":"e29579fa-0beb-4dea-8cde-bf5cd16afb5d","Type":"ContainerStarted","Data":"7e96a29e9439e0361b1688914adef84e54b33a520e973fe0884f37bda5858a69"} Oct 30 00:17:05.049935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:05.049197 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:17:05.067434 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:05.066865 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:17:05.067434 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:05.066964 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" podUID="e29579fa-0beb-4dea-8cde-bf5cd16afb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 00:17:05.078990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:05.078927 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" podStartSLOduration=2.078892145 podStartE2EDuration="2.078892145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:17:05.076386842 +0000 UTC m=+740.281888023" watchObservedRunningTime="2026-10-30 00:17:05.078892145 +0000 UTC m=+740.284393326" Oct 30 00:17:06.048357 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:06.048322 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:17:06.049090 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:06.049046 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" podUID="e29579fa-0beb-4dea-8cde-bf5cd16afb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 00:17:07.048382 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:07.048345 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:17:07.048773 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:07.048396 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" podUID="e29579fa-0beb-4dea-8cde-bf5cd16afb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 00:17:08.048352 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:08.048320 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:17:08.048559 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:08.048370 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" podUID="e29579fa-0beb-4dea-8cde-bf5cd16afb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 00:17:09.055934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:09.052323 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:17:09.055934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:09.052396 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" podUID="e29579fa-0beb-4dea-8cde-bf5cd16afb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 00:17:10.051939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:10.051329 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:17:10.051939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:10.051390 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" podUID="e29579fa-0beb-4dea-8cde-bf5cd16afb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 00:17:10.892101 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:10.892060 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d30f24e5-h88w4"] Oct 30 00:17:10.898334 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:10.898306 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30f24e5-h88w4" Oct 30 00:17:10.923932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:10.918885 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:17:10.923932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:10.919039 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:17:10.923932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:10.919126 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:17:10.923932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:10.923093 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d30f24e5-h88w4"] Oct 30 00:17:10.923932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:10.923258 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:17:11.003001 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:11.002961 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b747aa-4fb0-4568-889c-40e44a2b5351-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d30f24e5-h88w4\" (UID: \"95b747aa-4fb0-4568-889c-40e44a2b5351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30f24e5-h88w4" Oct 30 00:17:11.003174 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:11.003089 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fstl\" (UniqueName: \"kubernetes.io/projected/95b747aa-4fb0-4568-889c-40e44a2b5351-kube-api-access-5fstl\") pod \"maas-api-revoke-sub-d30f24e5-h88w4\" (UID: \"95b747aa-4fb0-4568-889c-40e44a2b5351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30f24e5-h88w4" Oct 30 00:17:11.003220 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:11.003185 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b747aa-4fb0-4568-889c-40e44a2b5351-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d30f24e5-h88w4\" (UID: \"95b747aa-4fb0-4568-889c-40e44a2b5351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30f24e5-h88w4" Oct 30 00:17:11.105144 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:11.105105 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b747aa-4fb0-4568-889c-40e44a2b5351-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d30f24e5-h88w4\" (UID: \"95b747aa-4fb0-4568-889c-40e44a2b5351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30f24e5-h88w4" Oct 30 00:17:11.105344 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:11.105192 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b747aa-4fb0-4568-889c-40e44a2b5351-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d30f24e5-h88w4\" (UID: \"95b747aa-4fb0-4568-889c-40e44a2b5351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30f24e5-h88w4" Oct 30 00:17:11.105344 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:11.105280 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fstl\" (UniqueName: \"kubernetes.io/projected/95b747aa-4fb0-4568-889c-40e44a2b5351-kube-api-access-5fstl\") pod \"maas-api-revoke-sub-d30f24e5-h88w4\" (UID: \"95b747aa-4fb0-4568-889c-40e44a2b5351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30f24e5-h88w4" Oct 30 00:17:11.105867 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:11.105844 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b747aa-4fb0-4568-889c-40e44a2b5351-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d30f24e5-h88w4\" (UID: \"95b747aa-4fb0-4568-889c-40e44a2b5351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30f24e5-h88w4" Oct 30 00:17:11.107523 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:11.107501 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b747aa-4fb0-4568-889c-40e44a2b5351-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d30f24e5-h88w4\" (UID: \"95b747aa-4fb0-4568-889c-40e44a2b5351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30f24e5-h88w4" Oct 30 00:17:11.126222 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:11.126167 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fstl\" (UniqueName: \"kubernetes.io/projected/95b747aa-4fb0-4568-889c-40e44a2b5351-kube-api-access-5fstl\") pod \"maas-api-revoke-sub-d30f24e5-h88w4\" (UID: \"95b747aa-4fb0-4568-889c-40e44a2b5351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30f24e5-h88w4" Oct 30 00:17:11.215596 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:11.215522 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30f24e5-h88w4" Oct 30 00:17:11.516871 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:11.516821 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d30f24e5-h88w4"] Oct 30 00:17:11.738730 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:11.738698 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:17:11.739188 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:11.739167 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:17:11.739891 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:11.739873 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:17:12.093178 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:12.092958 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30f24e5-h88w4" event={"ID":"95b747aa-4fb0-4568-889c-40e44a2b5351","Type":"ContainerStarted","Data":"3c9847498b93bbcd85e141dfa00b00ff8d0c939c1b9272e3e90d1f664b0bae5a"} Oct 30 00:17:12.093178 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:12.093022 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30f24e5-h88w4" event={"ID":"95b747aa-4fb0-4568-889c-40e44a2b5351","Type":"ContainerStarted","Data":"84a98f69b86ac40d745092d729d74873dae3335b26a9d072f7799b1a79f1a71d"} Oct 30 00:17:12.130348 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:12.130295 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30f24e5-h88w4" podStartSLOduration=2.130281672 podStartE2EDuration="2.130281672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:17:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:17:12.127179776 +0000 UTC m=+747.332680976" watchObservedRunningTime="2026-10-30 00:17:12.130281672 +0000 UTC m=+747.335782853" Oct 30 00:17:13.101436 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:13.101404 5001 generic.go:356] "Generic (PLEG): container finished" podID="95b747aa-4fb0-4568-889c-40e44a2b5351" containerID="3c9847498b93bbcd85e141dfa00b00ff8d0c939c1b9272e3e90d1f664b0bae5a" exitCode=0 Oct 30 00:17:13.101923 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:13.101486 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30f24e5-h88w4" event={"ID":"95b747aa-4fb0-4568-889c-40e44a2b5351","Type":"ContainerDied","Data":"3c9847498b93bbcd85e141dfa00b00ff8d0c939c1b9272e3e90d1f664b0bae5a"} Oct 30 00:17:14.256075 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:14.256053 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30f24e5-h88w4" Oct 30 00:17:14.341630 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:14.341593 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95b747aa-4fb0-4568-889c-40e44a2b5351-kube-api-access-5fstl\" (UniqueName: \"kubernetes.io/projected/95b747aa-4fb0-4568-889c-40e44a2b5351-kube-api-access-5fstl\") pod \"95b747aa-4fb0-4568-889c-40e44a2b5351\" (UID: \"95b747aa-4fb0-4568-889c-40e44a2b5351\") " Oct 30 00:17:14.341797 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:14.341703 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95b747aa-4fb0-4568-889c-40e44a2b5351-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b747aa-4fb0-4568-889c-40e44a2b5351-maas-api-xks-ca\") pod \"95b747aa-4fb0-4568-889c-40e44a2b5351\" (UID: \"95b747aa-4fb0-4568-889c-40e44a2b5351\") " Oct 30 00:17:14.341797 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:14.341743 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95b747aa-4fb0-4568-889c-40e44a2b5351-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b747aa-4fb0-4568-889c-40e44a2b5351-maas-api-openshift-ca\") pod \"95b747aa-4fb0-4568-889c-40e44a2b5351\" (UID: \"95b747aa-4fb0-4568-889c-40e44a2b5351\") " Oct 30 00:17:14.342188 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:14.342149 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95b747aa-4fb0-4568-889c-40e44a2b5351-maas-api-openshift-ca" pod "95b747aa-4fb0-4568-889c-40e44a2b5351" (UID: "95b747aa-4fb0-4568-889c-40e44a2b5351"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:17:14.343619 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:14.343597 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95b747aa-4fb0-4568-889c-40e44a2b5351-kube-api-access-5fstl" pod "95b747aa-4fb0-4568-889c-40e44a2b5351" (UID: "95b747aa-4fb0-4568-889c-40e44a2b5351"). InnerVolumeSpecName "kube-api-access-5fstl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:17:14.343825 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:14.343805 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95b747aa-4fb0-4568-889c-40e44a2b5351-maas-api-xks-ca" pod "95b747aa-4fb0-4568-889c-40e44a2b5351" (UID: "95b747aa-4fb0-4568-889c-40e44a2b5351"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:17:14.443329 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:14.443244 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fstl\" (UniqueName: \"kubernetes.io/projected/95b747aa-4fb0-4568-889c-40e44a2b5351-kube-api-access-5fstl\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:17:14.443329 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:14.443277 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b747aa-4fb0-4568-889c-40e44a2b5351-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:17:14.443329 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:14.443289 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b747aa-4fb0-4568-889c-40e44a2b5351-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:17:15.121608 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:15.121579 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30f24e5-h88w4" Oct 30 00:17:15.121754 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:15.121571 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30f24e5-h88w4" event={"ID":"95b747aa-4fb0-4568-889c-40e44a2b5351","Type":"ContainerDied","Data":"84a98f69b86ac40d745092d729d74873dae3335b26a9d072f7799b1a79f1a71d"} Oct 30 00:17:15.121754 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:15.121695 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84a98f69b86ac40d745092d729d74873dae3335b26a9d072f7799b1a79f1a71d" Oct 30 00:17:17.327352 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:17:17.327294 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:17:35.748878 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:35.748783 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s"] Oct 30 00:17:35.749526 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:35.749503 5001 cpu_manager.go:414] "containerMap: removing container" podUID="95b747aa-4fb0-4568-889c-40e44a2b5351" containerName="revoke-keys" Oct 30 00:17:35.749597 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:35.749526 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95b747aa-4fb0-4568-889c-40e44a2b5351" containerName="revoke-keys" Oct 30 00:17:35.749692 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:35.749679 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="95b747aa-4fb0-4568-889c-40e44a2b5351" containerName="revoke-keys" Oct 30 00:17:35.754583 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:35.754560 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:17:35.768203 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:35.768165 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-02665b\"/\"openshift-service-ca.crt\"" Oct 30 00:17:35.781945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:35.778216 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-02665b\"/\"distinct-w4-02665b-kserve-self-signed-certs\"" Oct 30 00:17:35.781945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:35.778370 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-02665b\"/\"default-dockercfg-tksv4\"" Oct 30 00:17:35.781945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:35.778447 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-02665b\"/\"kube-root-ca.crt\"" Oct 30 00:17:35.782690 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:35.782673 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s"] Oct 30 00:17:35.815966 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:35.815930 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-model-cache\") pod \"distinct-w4-02665b-kserve-84cb745cdd-gbp4s\" (UID: \"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:17:35.816131 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:35.816003 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-home\") pod \"distinct-w4-02665b-kserve-84cb745cdd-gbp4s\" (UID: \"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:17:35.816131 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:35.816058 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-tmp-dir\") pod \"distinct-w4-02665b-kserve-84cb745cdd-gbp4s\" (UID: \"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:17:35.816207 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:35.816154 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87nwg\" (UniqueName: \"kubernetes.io/projected/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-kube-api-access-87nwg\") pod \"distinct-w4-02665b-kserve-84cb745cdd-gbp4s\" (UID: \"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:17:35.816243 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:35.816232 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-dshm\") pod \"distinct-w4-02665b-kserve-84cb745cdd-gbp4s\" (UID: \"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:17:35.816396 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:35.816380 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-tls-certs\") pod \"distinct-w4-02665b-kserve-84cb745cdd-gbp4s\" (UID: \"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:17:35.917967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:35.917902 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-model-cache\") pod \"distinct-w4-02665b-kserve-84cb745cdd-gbp4s\" (UID: \"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:17:35.918159 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:35.917996 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-home\") pod \"distinct-w4-02665b-kserve-84cb745cdd-gbp4s\" (UID: \"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:17:35.918159 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:35.918046 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-tmp-dir\") pod \"distinct-w4-02665b-kserve-84cb745cdd-gbp4s\" (UID: \"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:17:35.918159 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:35.918086 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87nwg\" (UniqueName: \"kubernetes.io/projected/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-kube-api-access-87nwg\") pod \"distinct-w4-02665b-kserve-84cb745cdd-gbp4s\" (UID: \"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:17:35.918159 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:35.918141 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-dshm\") pod \"distinct-w4-02665b-kserve-84cb745cdd-gbp4s\" (UID: \"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:17:35.918376 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:35.918233 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-tls-certs\") pod \"distinct-w4-02665b-kserve-84cb745cdd-gbp4s\" (UID: \"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:17:35.918509 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:35.918481 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-home\") pod \"distinct-w4-02665b-kserve-84cb745cdd-gbp4s\" (UID: \"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:17:35.918711 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:35.918503 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-tmp-dir\") pod \"distinct-w4-02665b-kserve-84cb745cdd-gbp4s\" (UID: \"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:17:35.918830 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:35.918757 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-model-cache\") pod \"distinct-w4-02665b-kserve-84cb745cdd-gbp4s\" (UID: \"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:17:35.920520 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:35.920495 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-dshm\") pod \"distinct-w4-02665b-kserve-84cb745cdd-gbp4s\" (UID: \"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:17:35.920833 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:35.920809 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-tls-certs\") pod \"distinct-w4-02665b-kserve-84cb745cdd-gbp4s\" (UID: \"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:17:35.961254 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:35.961221 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87nwg\" (UniqueName: \"kubernetes.io/projected/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-kube-api-access-87nwg\") pod \"distinct-w4-02665b-kserve-84cb745cdd-gbp4s\" (UID: \"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9\") " pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:17:36.073493 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:36.073463 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:17:36.378376 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:36.378351 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s"] Oct 30 00:17:36.381662 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:17:36.381635 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3fa62b6_7b40_4b35_b3aa_926e1ab8bee9.slice/crio-2cd96961a82edf577cfc1af4c168393ce4ae94e5fffd6c50d462b12b847a5b5e WatchSource:0}: Error finding container 2cd96961a82edf577cfc1af4c168393ce4ae94e5fffd6c50d462b12b847a5b5e: Status 404 returned error can't find the container with id 2cd96961a82edf577cfc1af4c168393ce4ae94e5fffd6c50d462b12b847a5b5e Oct 30 00:17:37.310653 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:37.310607 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" event={"ID":"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9","Type":"ContainerStarted","Data":"ab6e9faeb5bd0521dc1e733d5ad31678e9a73e15086f6826894c076196c0570f"} Oct 30 00:17:37.311127 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:37.310660 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" event={"ID":"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9","Type":"ContainerStarted","Data":"2cd96961a82edf577cfc1af4c168393ce4ae94e5fffd6c50d462b12b847a5b5e"} Oct 30 00:17:37.348543 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:37.348486 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podStartSLOduration=2.348471476 podStartE2EDuration="2.348471476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:17:37.346219942 +0000 UTC m=+772.551721122" watchObservedRunningTime="2026-10-30 00:17:37.348471476 +0000 UTC m=+772.553972717" Oct 30 00:17:38.074580 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:38.074550 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:17:38.491762 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:38.491662 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13496902-jv9sj"] Oct 30 00:17:38.496552 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:38.496526 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13496902-jv9sj" Oct 30 00:17:38.515536 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:38.515509 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13496902-jv9sj"] Oct 30 00:17:38.518661 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:38.517712 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:17:38.518661 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:38.517718 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:17:38.518661 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:38.517752 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:17:38.523823 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:38.523794 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:17:38.549172 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:38.549141 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tbmc\" (UniqueName: \"kubernetes.io/projected/7d94a02d-a312-4c91-a1c7-bb09907fc5ae-kube-api-access-2tbmc\") pod \"maas-api-revoke-sub-13496902-jv9sj\" (UID: \"7d94a02d-a312-4c91-a1c7-bb09907fc5ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13496902-jv9sj" Oct 30 00:17:38.549320 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:38.549185 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d94a02d-a312-4c91-a1c7-bb09907fc5ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13496902-jv9sj\" (UID: \"7d94a02d-a312-4c91-a1c7-bb09907fc5ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13496902-jv9sj" Oct 30 00:17:38.549362 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:38.549340 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d94a02d-a312-4c91-a1c7-bb09907fc5ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13496902-jv9sj\" (UID: \"7d94a02d-a312-4c91-a1c7-bb09907fc5ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13496902-jv9sj" Oct 30 00:17:38.651029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:38.650983 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d94a02d-a312-4c91-a1c7-bb09907fc5ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13496902-jv9sj\" (UID: \"7d94a02d-a312-4c91-a1c7-bb09907fc5ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13496902-jv9sj" Oct 30 00:17:38.651237 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:38.651078 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tbmc\" (UniqueName: \"kubernetes.io/projected/7d94a02d-a312-4c91-a1c7-bb09907fc5ae-kube-api-access-2tbmc\") pod \"maas-api-revoke-sub-13496902-jv9sj\" (UID: \"7d94a02d-a312-4c91-a1c7-bb09907fc5ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13496902-jv9sj" Oct 30 00:17:38.651237 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:38.651108 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d94a02d-a312-4c91-a1c7-bb09907fc5ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13496902-jv9sj\" (UID: \"7d94a02d-a312-4c91-a1c7-bb09907fc5ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13496902-jv9sj" Oct 30 00:17:38.651638 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:38.651615 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d94a02d-a312-4c91-a1c7-bb09907fc5ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13496902-jv9sj\" (UID: \"7d94a02d-a312-4c91-a1c7-bb09907fc5ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13496902-jv9sj" Oct 30 00:17:38.653381 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:38.653357 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d94a02d-a312-4c91-a1c7-bb09907fc5ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13496902-jv9sj\" (UID: \"7d94a02d-a312-4c91-a1c7-bb09907fc5ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13496902-jv9sj" Oct 30 00:17:38.671800 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:38.671772 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tbmc\" (UniqueName: \"kubernetes.io/projected/7d94a02d-a312-4c91-a1c7-bb09907fc5ae-kube-api-access-2tbmc\") pod \"maas-api-revoke-sub-13496902-jv9sj\" (UID: \"7d94a02d-a312-4c91-a1c7-bb09907fc5ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13496902-jv9sj" Oct 30 00:17:38.824673 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:38.824641 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13496902-jv9sj" Oct 30 00:17:39.124527 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:39.124439 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13496902-jv9sj"] Oct 30 00:17:39.128019 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:17:39.127983 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d94a02d_a312_4c91_a1c7_bb09907fc5ae.slice/crio-ba10cdeb29e96930b5ca75ddd63f68ffd1ea3514a8f13615cbf26c8eae43e54d WatchSource:0}: Error finding container ba10cdeb29e96930b5ca75ddd63f68ffd1ea3514a8f13615cbf26c8eae43e54d: Status 404 returned error can't find the container with id ba10cdeb29e96930b5ca75ddd63f68ffd1ea3514a8f13615cbf26c8eae43e54d Oct 30 00:17:39.326275 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:39.326244 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13496902-jv9sj" event={"ID":"7d94a02d-a312-4c91-a1c7-bb09907fc5ae","Type":"ContainerStarted","Data":"ba10cdeb29e96930b5ca75ddd63f68ffd1ea3514a8f13615cbf26c8eae43e54d"} Oct 30 00:17:40.334311 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:40.334277 5001 generic.go:356] "Generic (PLEG): container finished" podID="7d94a02d-a312-4c91-a1c7-bb09907fc5ae" containerID="4ba564bf722c3c02baf113ffffee65efe90fb664a3b44694419e1ce7855df504" exitCode=0 Oct 30 00:17:40.334700 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:40.334319 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13496902-jv9sj" event={"ID":"7d94a02d-a312-4c91-a1c7-bb09907fc5ae","Type":"ContainerDied","Data":"4ba564bf722c3c02baf113ffffee65efe90fb664a3b44694419e1ce7855df504"} Oct 30 00:17:41.488746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:41.488721 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13496902-jv9sj" Oct 30 00:17:41.583844 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:41.583646 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d94a02d-a312-4c91-a1c7-bb09907fc5ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d94a02d-a312-4c91-a1c7-bb09907fc5ae-maas-api-xks-ca\") pod \"7d94a02d-a312-4c91-a1c7-bb09907fc5ae\" (UID: \"7d94a02d-a312-4c91-a1c7-bb09907fc5ae\") " Oct 30 00:17:41.584235 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:41.584207 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d94a02d-a312-4c91-a1c7-bb09907fc5ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d94a02d-a312-4c91-a1c7-bb09907fc5ae-maas-api-openshift-ca\") pod \"7d94a02d-a312-4c91-a1c7-bb09907fc5ae\" (UID: \"7d94a02d-a312-4c91-a1c7-bb09907fc5ae\") " Oct 30 00:17:41.584395 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:41.584379 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d94a02d-a312-4c91-a1c7-bb09907fc5ae-kube-api-access-2tbmc\" (UniqueName: \"kubernetes.io/projected/7d94a02d-a312-4c91-a1c7-bb09907fc5ae-kube-api-access-2tbmc\") pod \"7d94a02d-a312-4c91-a1c7-bb09907fc5ae\" (UID: \"7d94a02d-a312-4c91-a1c7-bb09907fc5ae\") " Oct 30 00:17:41.587357 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:41.587322 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d94a02d-a312-4c91-a1c7-bb09907fc5ae-maas-api-openshift-ca" pod "7d94a02d-a312-4c91-a1c7-bb09907fc5ae" (UID: "7d94a02d-a312-4c91-a1c7-bb09907fc5ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:17:41.593588 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:41.593529 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d94a02d-a312-4c91-a1c7-bb09907fc5ae-kube-api-access-2tbmc" pod "7d94a02d-a312-4c91-a1c7-bb09907fc5ae" (UID: "7d94a02d-a312-4c91-a1c7-bb09907fc5ae"). InnerVolumeSpecName "kube-api-access-2tbmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:17:41.597002 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:41.596896 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d94a02d-a312-4c91-a1c7-bb09907fc5ae-maas-api-xks-ca" pod "7d94a02d-a312-4c91-a1c7-bb09907fc5ae" (UID: "7d94a02d-a312-4c91-a1c7-bb09907fc5ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:17:41.685883 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:41.685848 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d94a02d-a312-4c91-a1c7-bb09907fc5ae-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:17:41.685883 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:41.685879 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d94a02d-a312-4c91-a1c7-bb09907fc5ae-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:17:41.685883 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:41.685889 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tbmc\" (UniqueName: \"kubernetes.io/projected/7d94a02d-a312-4c91-a1c7-bb09907fc5ae-kube-api-access-2tbmc\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:17:42.348211 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:42.348180 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13496902-jv9sj" Oct 30 00:17:42.348380 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:42.348212 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13496902-jv9sj" event={"ID":"7d94a02d-a312-4c91-a1c7-bb09907fc5ae","Type":"ContainerDied","Data":"ba10cdeb29e96930b5ca75ddd63f68ffd1ea3514a8f13615cbf26c8eae43e54d"} Oct 30 00:17:42.348380 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:42.348272 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba10cdeb29e96930b5ca75ddd63f68ffd1ea3514a8f13615cbf26c8eae43e54d" Oct 30 00:17:43.890242 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:43.890209 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm"] Oct 30 00:17:43.891020 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:43.890992 5001 cpu_manager.go:414] "containerMap: removing container" podUID="7d94a02d-a312-4c91-a1c7-bb09907fc5ae" containerName="revoke-keys" Oct 30 00:17:43.891020 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:43.891017 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d94a02d-a312-4c91-a1c7-bb09907fc5ae" containerName="revoke-keys" Oct 30 00:17:43.891188 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:43.891165 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d94a02d-a312-4c91-a1c7-bb09907fc5ae" containerName="revoke-keys" Oct 30 00:17:43.894776 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:43.894749 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:17:43.918435 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:43.918401 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:17:43.918616 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:43.918478 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:17:43.918616 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:43.918512 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922f38ba-e017-4005-9841-aa0fd46b1d3c-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:17:43.918616 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:43.918596 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:17:43.918795 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:43.918632 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922f38ba-e017-4005-9841-aa0fd46b1d3c-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:17:43.918795 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:43.918667 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:17:43.918795 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:43.918696 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdnl5\" (UniqueName: \"kubernetes.io/projected/922f38ba-e017-4005-9841-aa0fd46b1d3c-kube-api-access-kdnl5\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:17:43.918997 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:43.918829 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922f38ba-e017-4005-9841-aa0fd46b1d3c-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:17:43.918997 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:43.918884 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:17:43.922871 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:43.922843 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm"] Oct 30 00:17:44.020236 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:44.020186 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:17:44.020429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:44.020254 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:17:44.020429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:44.020282 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922f38ba-e017-4005-9841-aa0fd46b1d3c-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:17:44.020429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:44.020333 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:17:44.020429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:44.020362 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922f38ba-e017-4005-9841-aa0fd46b1d3c-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:17:44.020429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:44.020395 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:17:44.020429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:44.020422 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdnl5\" (UniqueName: \"kubernetes.io/projected/922f38ba-e017-4005-9841-aa0fd46b1d3c-kube-api-access-kdnl5\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:17:44.020791 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:44.020499 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922f38ba-e017-4005-9841-aa0fd46b1d3c-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:17:44.020791 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:44.020548 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:17:44.021051 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:44.021023 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:17:44.021424 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:44.021400 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:17:44.021878 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:44.021853 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:17:44.022013 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:44.021935 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:17:44.022413 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:44.022360 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922f38ba-e017-4005-9841-aa0fd46b1d3c-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:17:44.024201 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:44.024178 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:17:44.024827 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:44.024799 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922f38ba-e017-4005-9841-aa0fd46b1d3c-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:17:44.042832 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:44.042809 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdnl5\" (UniqueName: \"kubernetes.io/projected/922f38ba-e017-4005-9841-aa0fd46b1d3c-kube-api-access-kdnl5\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:17:44.043700 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:44.043678 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922f38ba-e017-4005-9841-aa0fd46b1d3c-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:17:44.209491 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:44.209407 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:17:44.525347 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:44.525317 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm"] Oct 30 00:17:44.527639 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:17:44.527614 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod922f38ba_e017_4005_9841_aa0fd46b1d3c.slice/crio-c8ce83852e80b2965abd13171c9f79b3d873805db3c301755d08002cb4aa25c7 WatchSource:0}: Error finding container c8ce83852e80b2965abd13171c9f79b3d873805db3c301755d08002cb4aa25c7: Status 404 returned error can't find the container with id c8ce83852e80b2965abd13171c9f79b3d873805db3c301755d08002cb4aa25c7 Oct 30 00:17:44.529955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:44.529902 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:17:44.530024 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:44.529989 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:17:44.530024 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:44.530018 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:17:45.370372 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:45.370318 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" event={"ID":"922f38ba-e017-4005-9841-aa0fd46b1d3c","Type":"ContainerStarted","Data":"b7fafa059445eb313e1005939bca2264a2ff067f01055e27dd2dc3fd1538a6a5"} Oct 30 00:17:45.370885 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:45.370426 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" event={"ID":"922f38ba-e017-4005-9841-aa0fd46b1d3c","Type":"ContainerStarted","Data":"c8ce83852e80b2965abd13171c9f79b3d873805db3c301755d08002cb4aa25c7"} Oct 30 00:17:45.420101 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:45.420046 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" podStartSLOduration=2.420032263 podStartE2EDuration="2.420032263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:17:45.418854965 +0000 UTC m=+780.624356146" watchObservedRunningTime="2026-10-30 00:17:45.420032263 +0000 UTC m=+780.625533440" Oct 30 00:17:46.073873 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:46.073837 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:17:46.087809 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:46.087780 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:17:46.215937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:46.213075 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:17:46.216150 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:46.215999 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 00:17:46.216150 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:46.216053 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" podUID="922f38ba-e017-4005-9841-aa0fd46b1d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 00:17:46.389173 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:46.389093 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:17:47.212937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:47.212292 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 00:17:47.212937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:47.212354 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" podUID="922f38ba-e017-4005-9841-aa0fd46b1d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 00:17:47.349097 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:17:47.349055 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:17:48.213765 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:48.213301 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 00:17:48.213765 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:48.213376 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" podUID="922f38ba-e017-4005-9841-aa0fd46b1d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 00:17:49.210320 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:49.210280 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 00:17:49.210621 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:49.210590 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" podUID="922f38ba-e017-4005-9841-aa0fd46b1d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 00:17:50.218056 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:50.218019 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 00:17:50.218636 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:50.218605 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" podUID="922f38ba-e017-4005-9841-aa0fd46b1d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 00:17:51.210573 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:51.210540 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 00:17:51.210746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:51.210591 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" podUID="922f38ba-e017-4005-9841-aa0fd46b1d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 00:17:52.214552 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:52.214520 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:17:52.214974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:52.214799 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:17:52.215545 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:52.215525 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:17:57.123938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:57.122637 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85"] Oct 30 00:17:57.129902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:57.129873 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:17:57.145846 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:57.145816 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85"] Oct 30 00:17:57.148736 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:57.148710 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-b1c718\"/\"openshift-service-ca.crt\"" Oct 30 00:17:57.148977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:57.148788 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-b1c718\"/\"default-dockercfg-q56cf\"" Oct 30 00:17:57.153105 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:57.153085 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-b1c718\"/\"distinct-2-w5-b1c718-kserve-self-signed-certs\"" Oct 30 00:17:57.153229 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:57.153211 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-b1c718\"/\"kube-root-ca.crt\"" Oct 30 00:17:57.209214 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:57.209178 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aabed96e-162a-4e58-a3b1-57750140a0e8-dshm\") pod \"distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85\" (UID: \"aabed96e-162a-4e58-a3b1-57750140a0e8\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:17:57.209214 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:57.209217 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aabed96e-162a-4e58-a3b1-57750140a0e8-tmp-dir\") pod \"distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85\" (UID: \"aabed96e-162a-4e58-a3b1-57750140a0e8\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:17:57.209473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:57.209270 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aabed96e-162a-4e58-a3b1-57750140a0e8-home\") pod \"distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85\" (UID: \"aabed96e-162a-4e58-a3b1-57750140a0e8\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:17:57.209473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:57.209363 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aabed96e-162a-4e58-a3b1-57750140a0e8-tls-certs\") pod \"distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85\" (UID: \"aabed96e-162a-4e58-a3b1-57750140a0e8\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:17:57.209574 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:57.209502 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aabed96e-162a-4e58-a3b1-57750140a0e8-model-cache\") pod \"distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85\" (UID: \"aabed96e-162a-4e58-a3b1-57750140a0e8\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:17:57.209574 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:57.209547 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlvx5\" (UniqueName: \"kubernetes.io/projected/aabed96e-162a-4e58-a3b1-57750140a0e8-kube-api-access-vlvx5\") pod \"distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85\" (UID: \"aabed96e-162a-4e58-a3b1-57750140a0e8\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:17:57.311666 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:57.311623 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aabed96e-162a-4e58-a3b1-57750140a0e8-tls-certs\") pod \"distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85\" (UID: \"aabed96e-162a-4e58-a3b1-57750140a0e8\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:17:57.312604 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:57.312575 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aabed96e-162a-4e58-a3b1-57750140a0e8-model-cache\") pod \"distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85\" (UID: \"aabed96e-162a-4e58-a3b1-57750140a0e8\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:17:57.320235 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:57.320207 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlvx5\" (UniqueName: \"kubernetes.io/projected/aabed96e-162a-4e58-a3b1-57750140a0e8-kube-api-access-vlvx5\") pod \"distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85\" (UID: \"aabed96e-162a-4e58-a3b1-57750140a0e8\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:17:57.320837 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:57.320816 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aabed96e-162a-4e58-a3b1-57750140a0e8-dshm\") pod \"distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85\" (UID: \"aabed96e-162a-4e58-a3b1-57750140a0e8\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:17:57.327943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:57.321346 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aabed96e-162a-4e58-a3b1-57750140a0e8-tmp-dir\") pod \"distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85\" (UID: \"aabed96e-162a-4e58-a3b1-57750140a0e8\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:17:57.328727 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:57.328705 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aabed96e-162a-4e58-a3b1-57750140a0e8-home\") pod \"distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85\" (UID: \"aabed96e-162a-4e58-a3b1-57750140a0e8\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:17:57.329287 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:57.320117 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aabed96e-162a-4e58-a3b1-57750140a0e8-model-cache\") pod \"distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85\" (UID: \"aabed96e-162a-4e58-a3b1-57750140a0e8\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:17:57.329431 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:57.328564 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aabed96e-162a-4e58-a3b1-57750140a0e8-tmp-dir\") pod \"distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85\" (UID: \"aabed96e-162a-4e58-a3b1-57750140a0e8\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:17:57.332117 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:57.329165 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aabed96e-162a-4e58-a3b1-57750140a0e8-home\") pod \"distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85\" (UID: \"aabed96e-162a-4e58-a3b1-57750140a0e8\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:17:57.333741 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:57.333715 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aabed96e-162a-4e58-a3b1-57750140a0e8-tls-certs\") pod \"distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85\" (UID: \"aabed96e-162a-4e58-a3b1-57750140a0e8\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:17:57.356647 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:57.356615 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlvx5\" (UniqueName: \"kubernetes.io/projected/aabed96e-162a-4e58-a3b1-57750140a0e8-kube-api-access-vlvx5\") pod \"distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85\" (UID: \"aabed96e-162a-4e58-a3b1-57750140a0e8\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:17:57.370262 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:57.370202 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aabed96e-162a-4e58-a3b1-57750140a0e8-dshm\") pod \"distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85\" (UID: \"aabed96e-162a-4e58-a3b1-57750140a0e8\") " pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:17:57.450929 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:57.450820 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:17:57.777484 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:57.777456 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85"] Oct 30 00:17:57.779866 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:17:57.779843 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaabed96e_162a_4e58_a3b1_57750140a0e8.slice/crio-2a76875f98957de06abb712ad0e7e21e1020f8b6acaebd4426cf7127b6a40e79 WatchSource:0}: Error finding container 2a76875f98957de06abb712ad0e7e21e1020f8b6acaebd4426cf7127b6a40e79: Status 404 returned error can't find the container with id 2a76875f98957de06abb712ad0e7e21e1020f8b6acaebd4426cf7127b6a40e79 Oct 30 00:17:58.496972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:58.496929 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" event={"ID":"aabed96e-162a-4e58-a3b1-57750140a0e8","Type":"ContainerStarted","Data":"03bebce622ce78c29cf07a2e18e5c4ff70786e27d839711ea040730c3ca8fd8b"} Oct 30 00:17:58.497361 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:58.496978 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" event={"ID":"aabed96e-162a-4e58-a3b1-57750140a0e8","Type":"ContainerStarted","Data":"2a76875f98957de06abb712ad0e7e21e1020f8b6acaebd4426cf7127b6a40e79"} Oct 30 00:17:58.536091 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:58.536037 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podStartSLOduration=1.5360228820000001 podStartE2EDuration="1.536022882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:17:58.532767459 +0000 UTC m=+793.738268640" watchObservedRunningTime="2026-10-30 00:17:58.536022882 +0000 UTC m=+793.741524063" Oct 30 00:17:59.452018 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:17:59.451987 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:18:00.458282 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:00.458253 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn"] Oct 30 00:18:00.462315 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:00.462294 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:18:00.470053 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:00.470022 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44fc62e1-bb2f-492a-af37-f412726dfb7f-tmp-dir\") pod \"distinct-w3-564c8f-kserve-6756c6b986-x6tnn\" (UID: \"44fc62e1-bb2f-492a-af37-f412726dfb7f\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:18:00.470211 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:00.470066 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44fc62e1-bb2f-492a-af37-f412726dfb7f-tls-certs\") pod \"distinct-w3-564c8f-kserve-6756c6b986-x6tnn\" (UID: \"44fc62e1-bb2f-492a-af37-f412726dfb7f\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:18:00.470211 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:00.470133 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44fc62e1-bb2f-492a-af37-f412726dfb7f-dshm\") pod \"distinct-w3-564c8f-kserve-6756c6b986-x6tnn\" (UID: \"44fc62e1-bb2f-492a-af37-f412726dfb7f\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:18:00.470211 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:00.470160 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44fc62e1-bb2f-492a-af37-f412726dfb7f-model-cache\") pod \"distinct-w3-564c8f-kserve-6756c6b986-x6tnn\" (UID: \"44fc62e1-bb2f-492a-af37-f412726dfb7f\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:18:00.470352 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:00.470220 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfp94\" (UniqueName: \"kubernetes.io/projected/44fc62e1-bb2f-492a-af37-f412726dfb7f-kube-api-access-cfp94\") pod \"distinct-w3-564c8f-kserve-6756c6b986-x6tnn\" (UID: \"44fc62e1-bb2f-492a-af37-f412726dfb7f\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:18:00.470352 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:00.470273 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44fc62e1-bb2f-492a-af37-f412726dfb7f-home\") pod \"distinct-w3-564c8f-kserve-6756c6b986-x6tnn\" (UID: \"44fc62e1-bb2f-492a-af37-f412726dfb7f\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:18:00.484786 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:00.484756 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn"] Oct 30 00:18:00.486400 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:00.486376 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-564c8f\"/\"openshift-service-ca.crt\"" Oct 30 00:18:00.486521 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:00.486411 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-564c8f\"/\"default-dockercfg-jqzgw\"" Oct 30 00:18:00.486590 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:00.486540 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-564c8f\"/\"kube-root-ca.crt\"" Oct 30 00:18:00.486590 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:00.486542 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-564c8f\"/\"distinct-w3-564c8f-kserve-self-signed-certs\"" Oct 30 00:18:00.570782 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:00.570744 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44fc62e1-bb2f-492a-af37-f412726dfb7f-home\") pod \"distinct-w3-564c8f-kserve-6756c6b986-x6tnn\" (UID: \"44fc62e1-bb2f-492a-af37-f412726dfb7f\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:18:00.570990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:00.570822 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44fc62e1-bb2f-492a-af37-f412726dfb7f-tmp-dir\") pod \"distinct-w3-564c8f-kserve-6756c6b986-x6tnn\" (UID: \"44fc62e1-bb2f-492a-af37-f412726dfb7f\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:18:00.570990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:00.570839 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44fc62e1-bb2f-492a-af37-f412726dfb7f-tls-certs\") pod \"distinct-w3-564c8f-kserve-6756c6b986-x6tnn\" (UID: \"44fc62e1-bb2f-492a-af37-f412726dfb7f\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:18:00.570990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:00.570897 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44fc62e1-bb2f-492a-af37-f412726dfb7f-dshm\") pod \"distinct-w3-564c8f-kserve-6756c6b986-x6tnn\" (UID: \"44fc62e1-bb2f-492a-af37-f412726dfb7f\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:18:00.570990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:00.570943 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44fc62e1-bb2f-492a-af37-f412726dfb7f-model-cache\") pod \"distinct-w3-564c8f-kserve-6756c6b986-x6tnn\" (UID: \"44fc62e1-bb2f-492a-af37-f412726dfb7f\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:18:00.571214 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:00.571008 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfp94\" (UniqueName: \"kubernetes.io/projected/44fc62e1-bb2f-492a-af37-f412726dfb7f-kube-api-access-cfp94\") pod \"distinct-w3-564c8f-kserve-6756c6b986-x6tnn\" (UID: \"44fc62e1-bb2f-492a-af37-f412726dfb7f\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:18:00.571274 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:00.571235 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44fc62e1-bb2f-492a-af37-f412726dfb7f-home\") pod \"distinct-w3-564c8f-kserve-6756c6b986-x6tnn\" (UID: \"44fc62e1-bb2f-492a-af37-f412726dfb7f\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:18:00.571326 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:00.571293 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44fc62e1-bb2f-492a-af37-f412726dfb7f-tmp-dir\") pod \"distinct-w3-564c8f-kserve-6756c6b986-x6tnn\" (UID: \"44fc62e1-bb2f-492a-af37-f412726dfb7f\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:18:00.571489 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:00.571473 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44fc62e1-bb2f-492a-af37-f412726dfb7f-model-cache\") pod \"distinct-w3-564c8f-kserve-6756c6b986-x6tnn\" (UID: \"44fc62e1-bb2f-492a-af37-f412726dfb7f\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:18:00.573429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:00.573401 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44fc62e1-bb2f-492a-af37-f412726dfb7f-dshm\") pod \"distinct-w3-564c8f-kserve-6756c6b986-x6tnn\" (UID: \"44fc62e1-bb2f-492a-af37-f412726dfb7f\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:18:00.573661 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:00.573645 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44fc62e1-bb2f-492a-af37-f412726dfb7f-tls-certs\") pod \"distinct-w3-564c8f-kserve-6756c6b986-x6tnn\" (UID: \"44fc62e1-bb2f-492a-af37-f412726dfb7f\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:18:00.591563 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:00.591535 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfp94\" (UniqueName: \"kubernetes.io/projected/44fc62e1-bb2f-492a-af37-f412726dfb7f-kube-api-access-cfp94\") pod \"distinct-w3-564c8f-kserve-6756c6b986-x6tnn\" (UID: \"44fc62e1-bb2f-492a-af37-f412726dfb7f\") " pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:18:00.784497 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:00.782116 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:18:01.114271 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:01.114163 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn"] Oct 30 00:18:01.116591 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:18:01.116564 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44fc62e1_bb2f_492a_af37_f412726dfb7f.slice/crio-f97c33f98f01a23ecab325013689ec43db9419bc9bde5876f5b9bcea28ff796d WatchSource:0}: Error finding container f97c33f98f01a23ecab325013689ec43db9419bc9bde5876f5b9bcea28ff796d: Status 404 returned error can't find the container with id f97c33f98f01a23ecab325013689ec43db9419bc9bde5876f5b9bcea28ff796d Oct 30 00:18:01.525555 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:01.525522 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" event={"ID":"44fc62e1-bb2f-492a-af37-f412726dfb7f","Type":"ContainerStarted","Data":"693cf671fd5479bf07b7de17356b2f2d2e8e643f67e2b1d405ef4677a1af11ce"} Oct 30 00:18:01.525949 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:01.525569 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" event={"ID":"44fc62e1-bb2f-492a-af37-f412726dfb7f","Type":"ContainerStarted","Data":"f97c33f98f01a23ecab325013689ec43db9419bc9bde5876f5b9bcea28ff796d"} Oct 30 00:18:01.567191 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:01.567138 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podStartSLOduration=1.567124979 podStartE2EDuration="1.567124979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:18:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:18:01.564396587 +0000 UTC m=+796.769897768" watchObservedRunningTime="2026-10-30 00:18:01.567124979 +0000 UTC m=+796.772626229" Oct 30 00:18:01.783360 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:01.783266 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:18:07.451769 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:07.451732 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:18:07.464618 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:07.464597 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:18:07.644825 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:07.644791 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:18:10.783556 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:10.783522 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:18:10.796185 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:10.796158 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:18:11.706206 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:11.706178 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:18:17.323475 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:18:17.323439 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:18:25.213022 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.212981 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6"] Oct 30 00:18:25.218570 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.218549 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:18:25.245590 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.245567 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6"] Oct 30 00:18:25.370825 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.370790 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2cdf918-8248-4079-9823-830a84e927bf-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:18:25.371050 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.370899 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:18:25.371050 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.370962 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2cdf918-8248-4079-9823-830a84e927bf-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:18:25.371050 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.370991 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:18:25.371050 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.371047 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:18:25.371266 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.371153 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjb4l\" (UniqueName: \"kubernetes.io/projected/b2cdf918-8248-4079-9823-830a84e927bf-kube-api-access-sjb4l\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:18:25.371266 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.371201 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:18:25.371366 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.371295 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:18:25.371366 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.371350 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2cdf918-8248-4079-9823-830a84e927bf-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:18:25.472577 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.472496 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:18:25.472728 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.472574 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:18:25.472728 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.472627 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2cdf918-8248-4079-9823-830a84e927bf-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:18:25.472728 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.472667 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2cdf918-8248-4079-9823-830a84e927bf-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:18:25.472873 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.472735 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:18:25.472873 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.472763 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2cdf918-8248-4079-9823-830a84e927bf-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:18:25.472873 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.472800 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:18:25.472873 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.472850 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:18:25.473103 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.472969 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjb4l\" (UniqueName: \"kubernetes.io/projected/b2cdf918-8248-4079-9823-830a84e927bf-kube-api-access-sjb4l\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:18:25.473103 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.473060 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:18:25.473222 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.473155 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:18:25.473295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.473271 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:18:25.473412 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.473317 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:18:25.473780 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.473760 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2cdf918-8248-4079-9823-830a84e927bf-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:18:25.475087 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.475065 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:18:25.475593 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.475570 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2cdf918-8248-4079-9823-830a84e927bf-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:18:25.496206 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.496183 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjb4l\" (UniqueName: \"kubernetes.io/projected/b2cdf918-8248-4079-9823-830a84e927bf-kube-api-access-sjb4l\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:18:25.496579 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.496557 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2cdf918-8248-4079-9823-830a84e927bf-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:18:25.534249 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.534222 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:18:25.852134 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.852104 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6"] Oct 30 00:18:25.853312 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:18:25.853283 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2cdf918_8248_4079_9823_830a84e927bf.slice/crio-95e3e4f929c19d924f2f2ab534bb99e7d515caf9719740a8b252c265f6765ddd WatchSource:0}: Error finding container 95e3e4f929c19d924f2f2ab534bb99e7d515caf9719740a8b252c265f6765ddd: Status 404 returned error can't find the container with id 95e3e4f929c19d924f2f2ab534bb99e7d515caf9719740a8b252c265f6765ddd Oct 30 00:18:25.855647 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.855613 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:18:25.855725 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.855680 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:18:25.855725 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:25.855711 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:18:26.838418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:26.838378 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" event={"ID":"b2cdf918-8248-4079-9823-830a84e927bf","Type":"ContainerStarted","Data":"69e51c74081b349c18d147927e847f86aedaf9f4774f2b7098fa342db180dc58"} Oct 30 00:18:26.838949 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:26.838442 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" event={"ID":"b2cdf918-8248-4079-9823-830a84e927bf","Type":"ContainerStarted","Data":"95e3e4f929c19d924f2f2ab534bb99e7d515caf9719740a8b252c265f6765ddd"} Oct 30 00:18:26.893202 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:26.893151 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" podStartSLOduration=1.893138257 podStartE2EDuration="1.893138257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:18:26.889889893 +0000 UTC m=+822.095391074" watchObservedRunningTime="2026-10-30 00:18:26.893138257 +0000 UTC m=+822.098639438" Oct 30 00:18:27.535300 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:27.535260 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:18:27.537184 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:27.537159 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 00:18:27.537311 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:27.537206 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" podUID="b2cdf918-8248-4079-9823-830a84e927bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 00:18:28.535936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:28.535614 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 00:18:28.535936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:28.535703 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" podUID="b2cdf918-8248-4079-9823-830a84e927bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 00:18:29.535340 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:29.535300 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 00:18:29.535546 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:29.535356 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" podUID="b2cdf918-8248-4079-9823-830a84e927bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 00:18:30.535689 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:30.535650 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 00:18:30.536121 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:30.535709 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" podUID="b2cdf918-8248-4079-9823-830a84e927bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 00:18:30.637637 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:30.637598 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-010356eb-zhrqb"] Oct 30 00:18:30.641603 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:30.641581 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-010356eb-zhrqb" Oct 30 00:18:30.666283 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:30.666252 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:18:30.666440 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:30.666283 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:18:30.666440 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:30.666349 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:18:30.666568 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:30.666510 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:18:30.670547 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:30.670521 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-010356eb-zhrqb"] Oct 30 00:18:30.756238 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:30.756202 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c92d1d-c016-49f5-a7ae-971c9702f505-maas-api-xks-ca\") pod \"maas-api-revoke-sub-010356eb-zhrqb\" (UID: \"a8c92d1d-c016-49f5-a7ae-971c9702f505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-010356eb-zhrqb" Oct 30 00:18:30.756406 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:30.756274 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4rgx\" (UniqueName: \"kubernetes.io/projected/a8c92d1d-c016-49f5-a7ae-971c9702f505-kube-api-access-r4rgx\") pod \"maas-api-revoke-sub-010356eb-zhrqb\" (UID: \"a8c92d1d-c016-49f5-a7ae-971c9702f505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-010356eb-zhrqb" Oct 30 00:18:30.756445 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:30.756424 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c92d1d-c016-49f5-a7ae-971c9702f505-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-010356eb-zhrqb\" (UID: \"a8c92d1d-c016-49f5-a7ae-971c9702f505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-010356eb-zhrqb" Oct 30 00:18:30.857319 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:30.857240 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c92d1d-c016-49f5-a7ae-971c9702f505-maas-api-xks-ca\") pod \"maas-api-revoke-sub-010356eb-zhrqb\" (UID: \"a8c92d1d-c016-49f5-a7ae-971c9702f505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-010356eb-zhrqb" Oct 30 00:18:30.857319 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:30.857278 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4rgx\" (UniqueName: \"kubernetes.io/projected/a8c92d1d-c016-49f5-a7ae-971c9702f505-kube-api-access-r4rgx\") pod \"maas-api-revoke-sub-010356eb-zhrqb\" (UID: \"a8c92d1d-c016-49f5-a7ae-971c9702f505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-010356eb-zhrqb" Oct 30 00:18:30.857514 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:30.857427 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c92d1d-c016-49f5-a7ae-971c9702f505-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-010356eb-zhrqb\" (UID: \"a8c92d1d-c016-49f5-a7ae-971c9702f505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-010356eb-zhrqb" Oct 30 00:18:30.858066 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:30.858045 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c92d1d-c016-49f5-a7ae-971c9702f505-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-010356eb-zhrqb\" (UID: \"a8c92d1d-c016-49f5-a7ae-971c9702f505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-010356eb-zhrqb" Oct 30 00:18:30.859572 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:30.859542 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c92d1d-c016-49f5-a7ae-971c9702f505-maas-api-xks-ca\") pod \"maas-api-revoke-sub-010356eb-zhrqb\" (UID: \"a8c92d1d-c016-49f5-a7ae-971c9702f505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-010356eb-zhrqb" Oct 30 00:18:30.877946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:30.877902 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4rgx\" (UniqueName: \"kubernetes.io/projected/a8c92d1d-c016-49f5-a7ae-971c9702f505-kube-api-access-r4rgx\") pod \"maas-api-revoke-sub-010356eb-zhrqb\" (UID: \"a8c92d1d-c016-49f5-a7ae-971c9702f505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-010356eb-zhrqb" Oct 30 00:18:30.956326 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:30.956293 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-010356eb-zhrqb" Oct 30 00:18:31.305338 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:31.305314 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-010356eb-zhrqb"] Oct 30 00:18:31.307836 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:18:31.307800 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c92d1d_c016_49f5_a7ae_971c9702f505.slice/crio-2481163211ef650827e741cacd311a46fdba98f170eca9efe374d6dec628c075 WatchSource:0}: Error finding container 2481163211ef650827e741cacd311a46fdba98f170eca9efe374d6dec628c075: Status 404 returned error can't find the container with id 2481163211ef650827e741cacd311a46fdba98f170eca9efe374d6dec628c075 Oct 30 00:18:31.535419 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:31.535367 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 00:18:31.535591 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:31.535449 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" podUID="b2cdf918-8248-4079-9823-830a84e927bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 00:18:31.901571 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:31.901533 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-010356eb-zhrqb" event={"ID":"a8c92d1d-c016-49f5-a7ae-971c9702f505","Type":"ContainerStarted","Data":"085fbd7be1965654c617b3218427b4b1439dd4769ff5a2b9682b36c3f69931b2"} Oct 30 00:18:31.901981 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:31.901593 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-010356eb-zhrqb" event={"ID":"a8c92d1d-c016-49f5-a7ae-971c9702f505","Type":"ContainerStarted","Data":"2481163211ef650827e741cacd311a46fdba98f170eca9efe374d6dec628c075"} Oct 30 00:18:31.937843 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:31.937786 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-010356eb-zhrqb" podStartSLOduration=1.937744091 podStartE2EDuration="1.937744091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:18:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:18:31.936374707 +0000 UTC m=+827.141875888" watchObservedRunningTime="2026-10-30 00:18:31.937744091 +0000 UTC m=+827.143245274" Oct 30 00:18:32.855166 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:32.855134 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:18:32.855591 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:32.855571 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:18:32.856450 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:32.856431 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:18:32.912277 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:32.912240 5001 generic.go:356] "Generic (PLEG): container finished" podID="a8c92d1d-c016-49f5-a7ae-971c9702f505" containerID="085fbd7be1965654c617b3218427b4b1439dd4769ff5a2b9682b36c3f69931b2" exitCode=0 Oct 30 00:18:32.922491 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:32.922444 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-010356eb-zhrqb" event={"ID":"a8c92d1d-c016-49f5-a7ae-971c9702f505","Type":"ContainerDied","Data":"085fbd7be1965654c617b3218427b4b1439dd4769ff5a2b9682b36c3f69931b2"} Oct 30 00:18:34.085313 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:34.085273 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-010356eb-zhrqb" Oct 30 00:18:34.227004 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:34.224667 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8c92d1d-c016-49f5-a7ae-971c9702f505-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c92d1d-c016-49f5-a7ae-971c9702f505-maas-api-xks-ca\") pod \"a8c92d1d-c016-49f5-a7ae-971c9702f505\" (UID: \"a8c92d1d-c016-49f5-a7ae-971c9702f505\") " Oct 30 00:18:34.227004 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:34.224727 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8c92d1d-c016-49f5-a7ae-971c9702f505-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c92d1d-c016-49f5-a7ae-971c9702f505-maas-api-openshift-ca\") pod \"a8c92d1d-c016-49f5-a7ae-971c9702f505\" (UID: \"a8c92d1d-c016-49f5-a7ae-971c9702f505\") " Oct 30 00:18:34.227004 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:34.224943 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8c92d1d-c016-49f5-a7ae-971c9702f505-kube-api-access-r4rgx\" (UniqueName: \"kubernetes.io/projected/a8c92d1d-c016-49f5-a7ae-971c9702f505-kube-api-access-r4rgx\") pod \"a8c92d1d-c016-49f5-a7ae-971c9702f505\" (UID: \"a8c92d1d-c016-49f5-a7ae-971c9702f505\") " Oct 30 00:18:34.249958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:34.239099 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c92d1d-c016-49f5-a7ae-971c9702f505-maas-api-openshift-ca" pod "a8c92d1d-c016-49f5-a7ae-971c9702f505" (UID: "a8c92d1d-c016-49f5-a7ae-971c9702f505"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:18:34.249958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:34.240062 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8c92d1d-c016-49f5-a7ae-971c9702f505-maas-api-xks-ca" pod "a8c92d1d-c016-49f5-a7ae-971c9702f505" (UID: "a8c92d1d-c016-49f5-a7ae-971c9702f505"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:18:34.262659 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:34.262625 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c92d1d-c016-49f5-a7ae-971c9702f505-kube-api-access-r4rgx" pod "a8c92d1d-c016-49f5-a7ae-971c9702f505" (UID: "a8c92d1d-c016-49f5-a7ae-971c9702f505"). InnerVolumeSpecName "kube-api-access-r4rgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:18:34.327045 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:34.327003 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4rgx\" (UniqueName: \"kubernetes.io/projected/a8c92d1d-c016-49f5-a7ae-971c9702f505-kube-api-access-r4rgx\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:18:34.327045 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:34.327048 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c92d1d-c016-49f5-a7ae-971c9702f505-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:18:34.327252 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:34.327062 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c92d1d-c016-49f5-a7ae-971c9702f505-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:18:34.927944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:34.927903 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-010356eb-zhrqb" Oct 30 00:18:34.928088 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:34.927939 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-010356eb-zhrqb" event={"ID":"a8c92d1d-c016-49f5-a7ae-971c9702f505","Type":"ContainerDied","Data":"2481163211ef650827e741cacd311a46fdba98f170eca9efe374d6dec628c075"} Oct 30 00:18:34.928088 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:34.927994 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2481163211ef650827e741cacd311a46fdba98f170eca9efe374d6dec628c075" Oct 30 00:18:37.088629 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.088594 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5"] Oct 30 00:18:37.089332 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.089308 5001 cpu_manager.go:414] "containerMap: removing container" podUID="a8c92d1d-c016-49f5-a7ae-971c9702f505" containerName="revoke-keys" Oct 30 00:18:37.089332 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.089329 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8c92d1d-c016-49f5-a7ae-971c9702f505" containerName="revoke-keys" Oct 30 00:18:37.089442 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.089430 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8c92d1d-c016-49f5-a7ae-971c9702f505" containerName="revoke-keys" Oct 30 00:18:37.093079 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.093060 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:18:37.102606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.102582 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw"] Oct 30 00:18:37.106407 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.106389 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:18:37.121327 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.121306 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5"] Oct 30 00:18:37.140676 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.139788 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw"] Oct 30 00:18:37.265656 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.265608 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:18:37.265903 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.265883 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:18:37.266157 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.266135 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:18:37.266380 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.266361 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:18:37.266556 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.266537 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:18:37.266756 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.266739 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f2a45ad-8a47-428c-b70f-515b50d98abd-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:18:37.266889 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.266872 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f2a45ad-8a47-428c-b70f-515b50d98abd-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:18:37.268946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.268923 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9q7m\" (UniqueName: \"kubernetes.io/projected/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-kube-api-access-z9q7m\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:18:37.269088 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.269072 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4684w\" (UniqueName: \"kubernetes.io/projected/3f2a45ad-8a47-428c-b70f-515b50d98abd-kube-api-access-4684w\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:18:37.269207 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.269191 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:18:37.269305 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.269291 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f2a45ad-8a47-428c-b70f-515b50d98abd-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:18:37.269401 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.269387 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:18:37.269666 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.269647 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:18:37.269847 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.269780 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:18:37.270013 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.269874 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:18:37.270013 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.269905 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:18:37.270013 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.269984 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:18:37.270241 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.270042 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:18:37.370806 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.370726 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:18:37.370806 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.370766 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:18:37.370806 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.370795 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:18:37.371102 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.370841 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:18:37.371102 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.370886 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:18:37.371102 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.370925 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:18:37.371102 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.370961 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:18:37.371102 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.370982 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:18:37.371102 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.371006 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:18:37.371408 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.371122 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f2a45ad-8a47-428c-b70f-515b50d98abd-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:18:37.371408 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.371164 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f2a45ad-8a47-428c-b70f-515b50d98abd-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:18:37.371408 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.371224 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9q7m\" (UniqueName: \"kubernetes.io/projected/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-kube-api-access-z9q7m\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:18:37.371408 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.371253 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4684w\" (UniqueName: \"kubernetes.io/projected/3f2a45ad-8a47-428c-b70f-515b50d98abd-kube-api-access-4684w\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:18:37.371408 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.371262 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:18:37.371408 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.371298 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:18:37.371408 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.371325 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f2a45ad-8a47-428c-b70f-515b50d98abd-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:18:37.371408 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.371351 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:18:37.371408 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.371391 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:18:37.371839 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.371431 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:18:37.371839 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.371526 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:18:37.371839 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.371538 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:18:37.371839 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.371774 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:18:37.372072 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.371852 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:18:37.372072 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.372028 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:18:37.372072 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.371228 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:18:37.372227 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.372138 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:18:37.372287 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.372246 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f2a45ad-8a47-428c-b70f-515b50d98abd-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:18:37.372343 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.372285 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:18:37.373839 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.373818 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:18:37.374351 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.374318 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:18:37.374521 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.374473 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:18:37.374602 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.374542 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f2a45ad-8a47-428c-b70f-515b50d98abd-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:18:37.391959 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.391903 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9q7m\" (UniqueName: \"kubernetes.io/projected/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-kube-api-access-z9q7m\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:18:37.392104 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.392055 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f2a45ad-8a47-428c-b70f-515b50d98abd-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:18:37.393443 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.393423 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:18:37.393760 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.393743 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4684w\" (UniqueName: \"kubernetes.io/projected/3f2a45ad-8a47-428c-b70f-515b50d98abd-kube-api-access-4684w\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:18:37.407795 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.407775 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:18:37.420229 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.420208 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:18:37.754137 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.754108 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5"] Oct 30 00:18:37.756820 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:18:37.756795 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod774a6da8_ca9f_4ca3_b132_9a449bb2d74c.slice/crio-08b5c48fe9ca8a126166a66bdf7182880a01cb1d1c9f2f6c5a4f417f25e9579f WatchSource:0}: Error finding container 08b5c48fe9ca8a126166a66bdf7182880a01cb1d1c9f2f6c5a4f417f25e9579f: Status 404 returned error can't find the container with id 08b5c48fe9ca8a126166a66bdf7182880a01cb1d1c9f2f6c5a4f417f25e9579f Oct 30 00:18:37.767503 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.759754 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw"] Oct 30 00:18:37.768681 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.768644 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:18:37.768792 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.768723 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:18:37.768792 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.768766 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:18:37.772611 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:18:37.772581 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f2a45ad_8a47_428c_b70f_515b50d98abd.slice/crio-515a21fc198f975b9478ba852b415564cdf78db03f54205833fd6853af2cf433 WatchSource:0}: Error finding container 515a21fc198f975b9478ba852b415564cdf78db03f54205833fd6853af2cf433: Status 404 returned error can't find the container with id 515a21fc198f975b9478ba852b415564cdf78db03f54205833fd6853af2cf433 Oct 30 00:18:37.785345 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.785315 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:18:37.785447 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.785387 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:18:37.785447 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.785432 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:18:37.955782 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.955325 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" event={"ID":"774a6da8-ca9f-4ca3-b132-9a449bb2d74c","Type":"ContainerStarted","Data":"53dc9b2a3e181c21c2532dd623eb0d739aabccee0de4de92297f2bead1638f3c"} Oct 30 00:18:37.955782 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.955387 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" event={"ID":"774a6da8-ca9f-4ca3-b132-9a449bb2d74c","Type":"ContainerStarted","Data":"08b5c48fe9ca8a126166a66bdf7182880a01cb1d1c9f2f6c5a4f417f25e9579f"} Oct 30 00:18:37.960882 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.960851 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" event={"ID":"3f2a45ad-8a47-428c-b70f-515b50d98abd","Type":"ContainerStarted","Data":"0fa1b578cb799bd148c2b287a38e31c32ae1d1d52c05a865c6e789893bc378ee"} Oct 30 00:18:37.961151 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.960898 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" event={"ID":"3f2a45ad-8a47-428c-b70f-515b50d98abd","Type":"ContainerStarted","Data":"515a21fc198f975b9478ba852b415564cdf78db03f54205833fd6853af2cf433"} Oct 30 00:18:37.972975 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.968698 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh"] Oct 30 00:18:37.973861 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.973842 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:18:37.994009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.987348 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-tmp-dir\") pod \"distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh\" (UID: \"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:18:37.994009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.987441 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-model-cache\") pod \"distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh\" (UID: \"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:18:37.994009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.987513 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-dshm\") pod \"distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh\" (UID: \"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:18:37.994009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.987528 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-tls-certs\") pod \"distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh\" (UID: \"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:18:37.994009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.987609 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmphn\" (UniqueName: \"kubernetes.io/projected/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-kube-api-access-dmphn\") pod \"distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh\" (UID: \"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:18:37.994009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.987735 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-home\") pod \"distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh\" (UID: \"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:18:37.994431 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.994031 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-7a7ac6\"/\"distinct-w1-7a7ac6-kserve-self-signed-certs\"" Oct 30 00:18:37.998533 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:37.998502 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh"] Oct 30 00:18:38.008475 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:38.008373 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" podStartSLOduration=1.008355931 podStartE2EDuration="1.008355931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:18:38.004591535 +0000 UTC m=+833.210092719" watchObservedRunningTime="2026-10-30 00:18:38.008355931 +0000 UTC m=+833.213857113" Oct 30 00:18:38.084206 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:38.084090 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" podStartSLOduration=1.084072118 podStartE2EDuration="1.084072118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:18:38.08145592 +0000 UTC m=+833.286957101" watchObservedRunningTime="2026-10-30 00:18:38.084072118 +0000 UTC m=+833.289573303" Oct 30 00:18:38.089358 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:38.089322 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-home\") pod \"distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh\" (UID: \"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:18:38.089771 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:38.089392 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-tmp-dir\") pod \"distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh\" (UID: \"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:18:38.089771 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:38.089487 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-model-cache\") pod \"distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh\" (UID: \"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:18:38.089771 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:38.089572 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-dshm\") pod \"distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh\" (UID: \"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:18:38.089771 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:38.089597 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-tls-certs\") pod \"distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh\" (UID: \"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:18:38.089771 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:38.089666 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmphn\" (UniqueName: \"kubernetes.io/projected/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-kube-api-access-dmphn\") pod \"distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh\" (UID: \"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:18:38.090438 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:38.090413 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-home\") pod \"distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh\" (UID: \"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:18:38.090956 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:38.090693 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-tmp-dir\") pod \"distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh\" (UID: \"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:18:38.090956 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:38.090906 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-model-cache\") pod \"distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh\" (UID: \"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:18:38.094183 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:38.094132 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-dshm\") pod \"distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh\" (UID: \"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:18:38.095640 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:38.095618 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-tls-certs\") pod \"distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh\" (UID: \"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:18:38.110351 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:38.110323 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmphn\" (UniqueName: \"kubernetes.io/projected/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-kube-api-access-dmphn\") pod \"distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh\" (UID: \"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:18:38.296980 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:38.295300 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:18:38.415384 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:38.409605 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:18:38.419952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:38.419597 5001 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 00:18:38.419952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:38.419650 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" podUID="774a6da8-ca9f-4ca3-b132-9a449bb2d74c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 00:18:38.432053 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:38.429797 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:18:38.445162 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:38.445034 5001 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 00:18:38.445162 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:38.445087 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" podUID="3f2a45ad-8a47-428c-b70f-515b50d98abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 00:18:38.647010 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:38.646563 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh"] Oct 30 00:18:38.971611 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:38.971517 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" event={"ID":"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f","Type":"ContainerStarted","Data":"82894cd7709ff0c84e160ae993e5f6ae2bb893241ba0f3a422140306490c4a1b"} Oct 30 00:18:38.971611 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:38.971579 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" event={"ID":"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f","Type":"ContainerStarted","Data":"70e158d2104c64a9bbd70419391851dd9effb43600f3838f10dabf89470b4faf"} Oct 30 00:18:39.029935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:39.010399 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podStartSLOduration=2.010380884 podStartE2EDuration="2.010380884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:18:39.007211945 +0000 UTC m=+834.212713125" watchObservedRunningTime="2026-10-30 00:18:39.010380884 +0000 UTC m=+834.215882066" Oct 30 00:18:39.296807 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:39.296776 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:18:39.411938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:39.410249 5001 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 00:18:39.411938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:39.410311 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" podUID="774a6da8-ca9f-4ca3-b132-9a449bb2d74c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 00:18:39.429937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:39.424334 5001 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 00:18:39.429937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:39.424395 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" podUID="3f2a45ad-8a47-428c-b70f-515b50d98abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 00:18:40.433941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:40.432499 5001 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 00:18:40.433941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:40.432557 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" podUID="3f2a45ad-8a47-428c-b70f-515b50d98abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 00:18:40.452177 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:40.452143 5001 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 00:18:40.452485 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:40.452456 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" podUID="774a6da8-ca9f-4ca3-b132-9a449bb2d74c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 00:18:40.966340 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:40.966311 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9c3d9cc-42dv5"] Oct 30 00:18:40.970727 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:40.970707 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c3d9cc-42dv5" Oct 30 00:18:40.989047 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:40.988899 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9c3d9cc-42dv5"] Oct 30 00:18:40.994582 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:40.994440 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:18:40.994582 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:40.994535 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:18:41.080465 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:41.080430 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:18:41.080633 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:41.080585 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:18:41.093679 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:41.093652 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcppr\" (UniqueName: \"kubernetes.io/projected/fb41aa3a-14f1-4213-af4b-80ced32a857a-kube-api-access-wcppr\") pod \"maas-api-revoke-sub-e9c3d9cc-42dv5\" (UID: \"fb41aa3a-14f1-4213-af4b-80ced32a857a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c3d9cc-42dv5" Oct 30 00:18:41.093862 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:41.093842 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb41aa3a-14f1-4213-af4b-80ced32a857a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9c3d9cc-42dv5\" (UID: \"fb41aa3a-14f1-4213-af4b-80ced32a857a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c3d9cc-42dv5" Oct 30 00:18:41.093955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:41.093907 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb41aa3a-14f1-4213-af4b-80ced32a857a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9c3d9cc-42dv5\" (UID: \"fb41aa3a-14f1-4213-af4b-80ced32a857a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c3d9cc-42dv5" Oct 30 00:18:41.195682 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:41.195636 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcppr\" (UniqueName: \"kubernetes.io/projected/fb41aa3a-14f1-4213-af4b-80ced32a857a-kube-api-access-wcppr\") pod \"maas-api-revoke-sub-e9c3d9cc-42dv5\" (UID: \"fb41aa3a-14f1-4213-af4b-80ced32a857a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c3d9cc-42dv5" Oct 30 00:18:41.195682 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:41.195687 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb41aa3a-14f1-4213-af4b-80ced32a857a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9c3d9cc-42dv5\" (UID: \"fb41aa3a-14f1-4213-af4b-80ced32a857a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c3d9cc-42dv5" Oct 30 00:18:41.195904 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:41.195734 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb41aa3a-14f1-4213-af4b-80ced32a857a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9c3d9cc-42dv5\" (UID: \"fb41aa3a-14f1-4213-af4b-80ced32a857a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c3d9cc-42dv5" Oct 30 00:18:41.196590 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:41.196562 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb41aa3a-14f1-4213-af4b-80ced32a857a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9c3d9cc-42dv5\" (UID: \"fb41aa3a-14f1-4213-af4b-80ced32a857a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c3d9cc-42dv5" Oct 30 00:18:41.198281 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:41.198263 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb41aa3a-14f1-4213-af4b-80ced32a857a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9c3d9cc-42dv5\" (UID: \"fb41aa3a-14f1-4213-af4b-80ced32a857a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c3d9cc-42dv5" Oct 30 00:18:41.236734 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:41.236652 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcppr\" (UniqueName: \"kubernetes.io/projected/fb41aa3a-14f1-4213-af4b-80ced32a857a-kube-api-access-wcppr\") pod \"maas-api-revoke-sub-e9c3d9cc-42dv5\" (UID: \"fb41aa3a-14f1-4213-af4b-80ced32a857a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c3d9cc-42dv5" Oct 30 00:18:41.286137 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:41.286109 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c3d9cc-42dv5" Oct 30 00:18:41.416820 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:41.416367 5001 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 00:18:41.416820 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:41.416427 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" podUID="774a6da8-ca9f-4ca3-b132-9a449bb2d74c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 00:18:41.421388 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:41.421258 5001 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 00:18:41.421388 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:41.421321 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" podUID="3f2a45ad-8a47-428c-b70f-515b50d98abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 00:18:41.664096 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:18:41.664065 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb41aa3a_14f1_4213_af4b_80ced32a857a.slice/crio-32b3fb69e0c243d8daa9d13870b842502b41fb79f57bcc9f75f422d69eb349e0 WatchSource:0}: Error finding container 32b3fb69e0c243d8daa9d13870b842502b41fb79f57bcc9f75f422d69eb349e0: Status 404 returned error can't find the container with id 32b3fb69e0c243d8daa9d13870b842502b41fb79f57bcc9f75f422d69eb349e0 Oct 30 00:18:41.667384 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:41.667342 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9c3d9cc-42dv5"] Oct 30 00:18:41.995732 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:41.995698 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c3d9cc-42dv5" event={"ID":"fb41aa3a-14f1-4213-af4b-80ced32a857a","Type":"ContainerStarted","Data":"32b3fb69e0c243d8daa9d13870b842502b41fb79f57bcc9f75f422d69eb349e0"} Oct 30 00:18:42.415949 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:42.412291 5001 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 00:18:42.415949 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:42.412344 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" podUID="774a6da8-ca9f-4ca3-b132-9a449bb2d74c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 00:18:42.425263 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:42.425230 5001 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 00:18:42.425496 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:42.425470 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" podUID="3f2a45ad-8a47-428c-b70f-515b50d98abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 00:18:43.004519 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:43.004435 5001 generic.go:356] "Generic (PLEG): container finished" podID="fb41aa3a-14f1-4213-af4b-80ced32a857a" containerID="e5956a44b8d39d61cd4e1d58cf00a2591f4f6ce5056fabfd31a6594245e20163" exitCode=0 Oct 30 00:18:43.004890 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:43.004635 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c3d9cc-42dv5" event={"ID":"fb41aa3a-14f1-4213-af4b-80ced32a857a","Type":"ContainerDied","Data":"e5956a44b8d39d61cd4e1d58cf00a2591f4f6ce5056fabfd31a6594245e20163"} Oct 30 00:18:43.416937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:43.412203 5001 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 00:18:43.416937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:43.412263 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" podUID="774a6da8-ca9f-4ca3-b132-9a449bb2d74c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 00:18:43.424936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:43.422252 5001 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 00:18:43.424936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:43.422311 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" podUID="3f2a45ad-8a47-428c-b70f-515b50d98abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 00:18:44.244227 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:44.244201 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c3d9cc-42dv5" Oct 30 00:18:44.374049 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:44.373960 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb41aa3a-14f1-4213-af4b-80ced32a857a-kube-api-access-wcppr\" (UniqueName: \"kubernetes.io/projected/fb41aa3a-14f1-4213-af4b-80ced32a857a-kube-api-access-wcppr\") pod \"fb41aa3a-14f1-4213-af4b-80ced32a857a\" (UID: \"fb41aa3a-14f1-4213-af4b-80ced32a857a\") " Oct 30 00:18:44.374217 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:44.374060 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb41aa3a-14f1-4213-af4b-80ced32a857a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb41aa3a-14f1-4213-af4b-80ced32a857a-maas-api-xks-ca\") pod \"fb41aa3a-14f1-4213-af4b-80ced32a857a\" (UID: \"fb41aa3a-14f1-4213-af4b-80ced32a857a\") " Oct 30 00:18:44.374217 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:44.374161 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb41aa3a-14f1-4213-af4b-80ced32a857a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb41aa3a-14f1-4213-af4b-80ced32a857a-maas-api-openshift-ca\") pod \"fb41aa3a-14f1-4213-af4b-80ced32a857a\" (UID: \"fb41aa3a-14f1-4213-af4b-80ced32a857a\") " Oct 30 00:18:44.374508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:44.374467 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb41aa3a-14f1-4213-af4b-80ced32a857a-maas-api-openshift-ca" pod "fb41aa3a-14f1-4213-af4b-80ced32a857a" (UID: "fb41aa3a-14f1-4213-af4b-80ced32a857a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:18:44.374810 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:44.374788 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb41aa3a-14f1-4213-af4b-80ced32a857a-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:18:44.376218 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:44.376194 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb41aa3a-14f1-4213-af4b-80ced32a857a-kube-api-access-wcppr" pod "fb41aa3a-14f1-4213-af4b-80ced32a857a" (UID: "fb41aa3a-14f1-4213-af4b-80ced32a857a"). InnerVolumeSpecName "kube-api-access-wcppr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:18:44.376638 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:44.376618 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb41aa3a-14f1-4213-af4b-80ced32a857a-maas-api-xks-ca" pod "fb41aa3a-14f1-4213-af4b-80ced32a857a" (UID: "fb41aa3a-14f1-4213-af4b-80ced32a857a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:18:44.408480 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:44.408451 5001 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 00:18:44.408642 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:44.408506 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" podUID="774a6da8-ca9f-4ca3-b132-9a449bb2d74c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 00:18:44.420790 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:44.420764 5001 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 00:18:44.420924 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:44.420808 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" podUID="3f2a45ad-8a47-428c-b70f-515b50d98abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 00:18:44.476142 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:44.476108 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcppr\" (UniqueName: \"kubernetes.io/projected/fb41aa3a-14f1-4213-af4b-80ced32a857a-kube-api-access-wcppr\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:18:44.476142 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:44.476141 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb41aa3a-14f1-4213-af4b-80ced32a857a-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:18:44.699564 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:44.699482 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9960d3c-fkjdx"] Oct 30 00:18:44.700341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:44.700320 5001 cpu_manager.go:414] "containerMap: removing container" podUID="fb41aa3a-14f1-4213-af4b-80ced32a857a" containerName="revoke-keys" Oct 30 00:18:44.700395 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:44.700347 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb41aa3a-14f1-4213-af4b-80ced32a857a" containerName="revoke-keys" Oct 30 00:18:44.700553 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:44.700540 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb41aa3a-14f1-4213-af4b-80ced32a857a" containerName="revoke-keys" Oct 30 00:18:44.704636 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:44.704612 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9960d3c-fkjdx" Oct 30 00:18:44.725759 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:44.724375 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9960d3c-fkjdx"] Oct 30 00:18:44.880891 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:44.880854 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5hgh\" (UniqueName: \"kubernetes.io/projected/b5bf89d2-732e-4a71-8ba3-7db0affe4155-kube-api-access-t5hgh\") pod \"maas-api-revoke-sub-d9960d3c-fkjdx\" (UID: \"b5bf89d2-732e-4a71-8ba3-7db0affe4155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9960d3c-fkjdx" Oct 30 00:18:44.881096 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:44.881068 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5bf89d2-732e-4a71-8ba3-7db0affe4155-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9960d3c-fkjdx\" (UID: \"b5bf89d2-732e-4a71-8ba3-7db0affe4155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9960d3c-fkjdx" Oct 30 00:18:44.881203 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:44.881186 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5bf89d2-732e-4a71-8ba3-7db0affe4155-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9960d3c-fkjdx\" (UID: \"b5bf89d2-732e-4a71-8ba3-7db0affe4155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9960d3c-fkjdx" Oct 30 00:18:44.983013 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:44.982901 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5bf89d2-732e-4a71-8ba3-7db0affe4155-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9960d3c-fkjdx\" (UID: \"b5bf89d2-732e-4a71-8ba3-7db0affe4155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9960d3c-fkjdx" Oct 30 00:18:44.983323 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:44.983292 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5bf89d2-732e-4a71-8ba3-7db0affe4155-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9960d3c-fkjdx\" (UID: \"b5bf89d2-732e-4a71-8ba3-7db0affe4155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9960d3c-fkjdx" Oct 30 00:18:44.983463 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:44.983446 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5hgh\" (UniqueName: \"kubernetes.io/projected/b5bf89d2-732e-4a71-8ba3-7db0affe4155-kube-api-access-t5hgh\") pod \"maas-api-revoke-sub-d9960d3c-fkjdx\" (UID: \"b5bf89d2-732e-4a71-8ba3-7db0affe4155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9960d3c-fkjdx" Oct 30 00:18:44.983785 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:44.983754 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5bf89d2-732e-4a71-8ba3-7db0affe4155-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9960d3c-fkjdx\" (UID: \"b5bf89d2-732e-4a71-8ba3-7db0affe4155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9960d3c-fkjdx" Oct 30 00:18:44.985890 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:44.985866 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5bf89d2-732e-4a71-8ba3-7db0affe4155-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9960d3c-fkjdx\" (UID: \"b5bf89d2-732e-4a71-8ba3-7db0affe4155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9960d3c-fkjdx" Oct 30 00:18:45.005174 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:45.005122 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5hgh\" (UniqueName: \"kubernetes.io/projected/b5bf89d2-732e-4a71-8ba3-7db0affe4155-kube-api-access-t5hgh\") pod \"maas-api-revoke-sub-d9960d3c-fkjdx\" (UID: \"b5bf89d2-732e-4a71-8ba3-7db0affe4155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9960d3c-fkjdx" Oct 30 00:18:45.026274 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:45.026023 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9960d3c-fkjdx" Oct 30 00:18:45.055105 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:45.055084 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c3d9cc-42dv5" Oct 30 00:18:45.055381 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:45.055312 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c3d9cc-42dv5" event={"ID":"fb41aa3a-14f1-4213-af4b-80ced32a857a","Type":"ContainerDied","Data":"32b3fb69e0c243d8daa9d13870b842502b41fb79f57bcc9f75f422d69eb349e0"} Oct 30 00:18:45.055381 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:45.055376 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32b3fb69e0c243d8daa9d13870b842502b41fb79f57bcc9f75f422d69eb349e0" Oct 30 00:18:45.396419 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:45.396330 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9960d3c-fkjdx"] Oct 30 00:18:45.604430 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:45.603986 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:18:45.607291 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:45.605497 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:18:45.607291 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:45.607018 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:18:45.693687 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:45.692736 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:18:45.693975 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:45.693951 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:18:45.695888 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:45.694897 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:18:46.064428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:46.064388 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9960d3c-fkjdx" event={"ID":"b5bf89d2-732e-4a71-8ba3-7db0affe4155","Type":"ContainerStarted","Data":"dbf10a479bf4c142cdaa352311d5a3ee182038547c48600b4ddc4d4e0964fcc7"} Oct 30 00:18:46.064597 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:46.064450 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9960d3c-fkjdx" event={"ID":"b5bf89d2-732e-4a71-8ba3-7db0affe4155","Type":"ContainerStarted","Data":"5f83aae1f8e646bb65b9f9c32d1d35b11e8728d4e71eb70907747e30e6d089a3"} Oct 30 00:18:46.102843 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:46.102784 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9960d3c-fkjdx" podStartSLOduration=2.102756234 podStartE2EDuration="2.102756234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:18:46.09852043 +0000 UTC m=+841.304021598" watchObservedRunningTime="2026-10-30 00:18:46.102756234 +0000 UTC m=+841.308257416" Oct 30 00:18:47.094143 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:47.093323 5001 generic.go:356] "Generic (PLEG): container finished" podID="b5bf89d2-732e-4a71-8ba3-7db0affe4155" containerID="dbf10a479bf4c142cdaa352311d5a3ee182038547c48600b4ddc4d4e0964fcc7" exitCode=0 Oct 30 00:18:47.095388 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:47.095202 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9960d3c-fkjdx" event={"ID":"b5bf89d2-732e-4a71-8ba3-7db0affe4155","Type":"ContainerDied","Data":"dbf10a479bf4c142cdaa352311d5a3ee182038547c48600b4ddc4d4e0964fcc7"} Oct 30 00:18:47.323388 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:18:47.323347 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:18:48.253326 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:48.253301 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9960d3c-fkjdx" Oct 30 00:18:48.296316 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:48.296286 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:18:48.309974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:48.309892 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:18:48.326890 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:48.326864 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5bf89d2-732e-4a71-8ba3-7db0affe4155-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5bf89d2-732e-4a71-8ba3-7db0affe4155-maas-api-xks-ca\") pod \"b5bf89d2-732e-4a71-8ba3-7db0affe4155\" (UID: \"b5bf89d2-732e-4a71-8ba3-7db0affe4155\") " Oct 30 00:18:48.327102 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:48.327084 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5bf89d2-732e-4a71-8ba3-7db0affe4155-kube-api-access-t5hgh\" (UniqueName: \"kubernetes.io/projected/b5bf89d2-732e-4a71-8ba3-7db0affe4155-kube-api-access-t5hgh\") pod \"b5bf89d2-732e-4a71-8ba3-7db0affe4155\" (UID: \"b5bf89d2-732e-4a71-8ba3-7db0affe4155\") " Oct 30 00:18:48.327177 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:48.327142 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5bf89d2-732e-4a71-8ba3-7db0affe4155-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5bf89d2-732e-4a71-8ba3-7db0affe4155-maas-api-openshift-ca\") pod \"b5bf89d2-732e-4a71-8ba3-7db0affe4155\" (UID: \"b5bf89d2-732e-4a71-8ba3-7db0affe4155\") " Oct 30 00:18:48.327612 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:48.327575 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5bf89d2-732e-4a71-8ba3-7db0affe4155-maas-api-openshift-ca" pod "b5bf89d2-732e-4a71-8ba3-7db0affe4155" (UID: "b5bf89d2-732e-4a71-8ba3-7db0affe4155"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:18:48.329192 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:48.329162 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5bf89d2-732e-4a71-8ba3-7db0affe4155-kube-api-access-t5hgh" pod "b5bf89d2-732e-4a71-8ba3-7db0affe4155" (UID: "b5bf89d2-732e-4a71-8ba3-7db0affe4155"). InnerVolumeSpecName "kube-api-access-t5hgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:18:48.329297 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:48.329194 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5bf89d2-732e-4a71-8ba3-7db0affe4155-maas-api-xks-ca" pod "b5bf89d2-732e-4a71-8ba3-7db0affe4155" (UID: "b5bf89d2-732e-4a71-8ba3-7db0affe4155"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:18:48.428528 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:48.428440 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5hgh\" (UniqueName: \"kubernetes.io/projected/b5bf89d2-732e-4a71-8ba3-7db0affe4155-kube-api-access-t5hgh\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:18:48.428528 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:48.428470 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5bf89d2-732e-4a71-8ba3-7db0affe4155-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:18:48.428528 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:48.428480 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5bf89d2-732e-4a71-8ba3-7db0affe4155-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:18:49.109096 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.109059 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9960d3c-fkjdx" event={"ID":"b5bf89d2-732e-4a71-8ba3-7db0affe4155","Type":"ContainerDied","Data":"5f83aae1f8e646bb65b9f9c32d1d35b11e8728d4e71eb70907747e30e6d089a3"} Oct 30 00:18:49.109274 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.109114 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f83aae1f8e646bb65b9f9c32d1d35b11e8728d4e71eb70907747e30e6d089a3" Oct 30 00:18:49.109274 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.109153 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9960d3c-fkjdx" Oct 30 00:18:49.121122 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.121098 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:18:49.729185 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.729147 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw"] Oct 30 00:18:49.729686 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.729671 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b5bf89d2-732e-4a71-8ba3-7db0affe4155" containerName="revoke-keys" Oct 30 00:18:49.729686 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.729685 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5bf89d2-732e-4a71-8ba3-7db0affe4155" containerName="revoke-keys" Oct 30 00:18:49.729813 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.729804 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5bf89d2-732e-4a71-8ba3-7db0affe4155" containerName="revoke-keys" Oct 30 00:18:49.741185 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.741157 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:18:49.742143 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.742125 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65d1bac8-837e-446b-9340-5fd94ba8be01-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:18:49.742232 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.742168 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:18:49.742232 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.742189 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:18:49.742334 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.742281 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65d1bac8-837e-446b-9340-5fd94ba8be01-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:18:49.742447 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.742424 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:18:49.742518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.742475 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:18:49.742562 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.742547 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8sbl\" (UniqueName: \"kubernetes.io/projected/65d1bac8-837e-446b-9340-5fd94ba8be01-kube-api-access-d8sbl\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:18:49.742609 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.742570 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:18:49.742696 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.742681 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65d1bac8-837e-446b-9340-5fd94ba8be01-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:18:49.760249 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.760223 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw"] Oct 30 00:18:49.766179 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.766153 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-322d20-openshift-default-dockercfg-5q64p\"" Oct 30 00:18:49.843782 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.843756 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:18:49.843931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.843791 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:18:49.843931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.843809 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65d1bac8-837e-446b-9340-5fd94ba8be01-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:18:49.843931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.843882 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:18:49.844106 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.843906 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:18:49.844106 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.843982 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8sbl\" (UniqueName: \"kubernetes.io/projected/65d1bac8-837e-446b-9340-5fd94ba8be01-kube-api-access-d8sbl\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:18:49.844106 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.844001 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:18:49.844106 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.844039 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65d1bac8-837e-446b-9340-5fd94ba8be01-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:18:49.844106 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.844103 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65d1bac8-837e-446b-9340-5fd94ba8be01-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:18:49.844357 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.844257 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:18:49.844357 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.844278 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:18:49.844357 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.844337 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:18:49.844543 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.844517 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:18:49.844839 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.844819 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65d1bac8-837e-446b-9340-5fd94ba8be01-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:18:49.846538 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.846519 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65d1bac8-837e-446b-9340-5fd94ba8be01-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:18:49.846752 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.846726 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:18:49.865761 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.865730 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8sbl\" (UniqueName: \"kubernetes.io/projected/65d1bac8-837e-446b-9340-5fd94ba8be01-kube-api-access-d8sbl\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:18:49.866763 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:49.866734 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65d1bac8-837e-446b-9340-5fd94ba8be01-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:18:50.102961 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:50.089156 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:18:50.450204 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:50.450176 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw"] Oct 30 00:18:50.451687 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:18:50.451654 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d1bac8_837e_446b_9340_5fd94ba8be01.slice/crio-8ccc8b325cda184872391d91eb7e09d8da4ea6cee8692e5accd3872e2d5aa7b1 WatchSource:0}: Error finding container 8ccc8b325cda184872391d91eb7e09d8da4ea6cee8692e5accd3872e2d5aa7b1: Status 404 returned error can't find the container with id 8ccc8b325cda184872391d91eb7e09d8da4ea6cee8692e5accd3872e2d5aa7b1 Oct 30 00:18:50.454175 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:50.454141 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:18:50.454265 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:50.454205 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:18:50.454265 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:50.454234 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:18:51.186500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:51.186455 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" event={"ID":"65d1bac8-837e-446b-9340-5fd94ba8be01","Type":"ContainerStarted","Data":"e8ec9fa3fefbbefeead300db070e78d1be471f973205314474bb1e50d53b0108"} Oct 30 00:18:51.187112 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:51.187072 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" event={"ID":"65d1bac8-837e-446b-9340-5fd94ba8be01","Type":"ContainerStarted","Data":"8ccc8b325cda184872391d91eb7e09d8da4ea6cee8692e5accd3872e2d5aa7b1"} Oct 30 00:18:51.269404 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:51.269339 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" podStartSLOduration=2.26931738 podStartE2EDuration="2.26931738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:18:51.267293511 +0000 UTC m=+846.472794716" watchObservedRunningTime="2026-10-30 00:18:51.26931738 +0000 UTC m=+846.474818563" Oct 30 00:18:52.095145 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:52.095114 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:18:52.095559 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:52.095533 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 00:18:52.095744 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:52.095720 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" podUID="65d1bac8-837e-446b-9340-5fd94ba8be01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 00:18:52.400359 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:52.400264 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 00:18:52.462113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:52.454287 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:18:52.690992 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:52.688017 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:18:52.715935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:52.705508 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:18:53.102940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:53.102685 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 00:18:53.102940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:53.102750 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" podUID="65d1bac8-837e-446b-9340-5fd94ba8be01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 00:18:53.105133 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:53.105103 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 00:18:53.139046 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:53.139001 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:18:53.470068 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:53.469970 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:18:53.471318 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:53.471294 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:18:54.177506 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:54.177474 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 00:18:54.181409 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:54.181370 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" podUID="65d1bac8-837e-446b-9340-5fd94ba8be01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 00:18:54.274064 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:54.274016 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:18:54.279442 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:54.279412 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:18:54.382048 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:54.381999 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:18:54.471227 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:54.471134 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:18:54.495333 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:54.495305 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:18:54.512638 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:54.512610 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:18:54.786106 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:54.786068 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cf9bf0c-sc5xw"] Oct 30 00:18:54.791287 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:54.791253 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cf9bf0c-sc5xw"] Oct 30 00:18:54.791457 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:54.791376 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf9bf0c-sc5xw" Oct 30 00:18:54.809159 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:54.809127 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:18:54.819901 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:54.814188 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:18:54.820210 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:54.818440 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:18:54.820336 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:54.818563 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:18:54.941186 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:54.941145 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0649729d-adb7-4ed6-90da-853e44f73236-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cf9bf0c-sc5xw\" (UID: \"0649729d-adb7-4ed6-90da-853e44f73236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf9bf0c-sc5xw" Oct 30 00:18:54.943478 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:54.943454 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0649729d-adb7-4ed6-90da-853e44f73236-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cf9bf0c-sc5xw\" (UID: \"0649729d-adb7-4ed6-90da-853e44f73236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf9bf0c-sc5xw" Oct 30 00:18:54.943875 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:54.943854 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgbch\" (UniqueName: \"kubernetes.io/projected/0649729d-adb7-4ed6-90da-853e44f73236-kube-api-access-zgbch\") pod \"maas-api-revoke-sub-9cf9bf0c-sc5xw\" (UID: \"0649729d-adb7-4ed6-90da-853e44f73236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf9bf0c-sc5xw" Oct 30 00:18:55.045812 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:55.045712 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgbch\" (UniqueName: \"kubernetes.io/projected/0649729d-adb7-4ed6-90da-853e44f73236-kube-api-access-zgbch\") pod \"maas-api-revoke-sub-9cf9bf0c-sc5xw\" (UID: \"0649729d-adb7-4ed6-90da-853e44f73236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf9bf0c-sc5xw" Oct 30 00:18:55.045812 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:55.045796 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0649729d-adb7-4ed6-90da-853e44f73236-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cf9bf0c-sc5xw\" (UID: \"0649729d-adb7-4ed6-90da-853e44f73236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf9bf0c-sc5xw" Oct 30 00:18:55.046126 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:55.045850 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0649729d-adb7-4ed6-90da-853e44f73236-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cf9bf0c-sc5xw\" (UID: \"0649729d-adb7-4ed6-90da-853e44f73236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf9bf0c-sc5xw" Oct 30 00:18:55.047019 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:55.046991 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0649729d-adb7-4ed6-90da-853e44f73236-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cf9bf0c-sc5xw\" (UID: \"0649729d-adb7-4ed6-90da-853e44f73236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf9bf0c-sc5xw" Oct 30 00:18:55.048239 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:55.048214 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0649729d-adb7-4ed6-90da-853e44f73236-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cf9bf0c-sc5xw\" (UID: \"0649729d-adb7-4ed6-90da-853e44f73236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf9bf0c-sc5xw" Oct 30 00:18:55.066600 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:55.066568 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgbch\" (UniqueName: \"kubernetes.io/projected/0649729d-adb7-4ed6-90da-853e44f73236-kube-api-access-zgbch\") pod \"maas-api-revoke-sub-9cf9bf0c-sc5xw\" (UID: \"0649729d-adb7-4ed6-90da-853e44f73236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf9bf0c-sc5xw" Oct 30 00:18:55.089992 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:55.089967 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 00:18:55.090101 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:55.090019 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" podUID="65d1bac8-837e-446b-9340-5fd94ba8be01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 00:18:55.121150 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:55.121115 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf9bf0c-sc5xw" Oct 30 00:18:55.440873 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:55.440841 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cf9bf0c-sc5xw"] Oct 30 00:18:55.443066 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:18:55.443037 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0649729d_adb7_4ed6_90da_853e44f73236.slice/crio-d277b00bc35b56a6904fe12c45de0a6516a79b19fbd65f1dbcc7fbeb8fa94034 WatchSource:0}: Error finding container d277b00bc35b56a6904fe12c45de0a6516a79b19fbd65f1dbcc7fbeb8fa94034: Status 404 returned error can't find the container with id d277b00bc35b56a6904fe12c45de0a6516a79b19fbd65f1dbcc7fbeb8fa94034 Oct 30 00:18:55.937967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:55.937471 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf9bf0c-sc5xw" event={"ID":"0649729d-adb7-4ed6-90da-853e44f73236","Type":"ContainerStarted","Data":"94cd6aa26317edd3d8e0e017c599b9a628fb9e960436c76b7932a3f04ed77406"} Oct 30 00:18:55.937967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:55.937546 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf9bf0c-sc5xw" event={"ID":"0649729d-adb7-4ed6-90da-853e44f73236","Type":"ContainerStarted","Data":"d277b00bc35b56a6904fe12c45de0a6516a79b19fbd65f1dbcc7fbeb8fa94034"} Oct 30 00:18:56.012931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:56.011003 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf9bf0c-sc5xw" podStartSLOduration=2.010979037 podStartE2EDuration="2.010979037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:18:55.991272503 +0000 UTC m=+851.196773685" watchObservedRunningTime="2026-10-30 00:18:56.010979037 +0000 UTC m=+851.216480231" Oct 30 00:18:56.090332 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:56.090258 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 00:18:56.090466 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:56.090320 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" podUID="65d1bac8-837e-446b-9340-5fd94ba8be01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 00:18:56.945559 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:56.945467 5001 generic.go:356] "Generic (PLEG): container finished" podID="0649729d-adb7-4ed6-90da-853e44f73236" containerID="94cd6aa26317edd3d8e0e017c599b9a628fb9e960436c76b7932a3f04ed77406" exitCode=0 Oct 30 00:18:56.945559 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:56.945540 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf9bf0c-sc5xw" event={"ID":"0649729d-adb7-4ed6-90da-853e44f73236","Type":"ContainerDied","Data":"94cd6aa26317edd3d8e0e017c599b9a628fb9e960436c76b7932a3f04ed77406"} Oct 30 00:18:57.090640 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:57.090608 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 00:18:57.090802 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:57.090659 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" podUID="65d1bac8-837e-446b-9340-5fd94ba8be01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 00:18:58.090388 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:58.090352 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 00:18:58.090864 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:58.090417 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" podUID="65d1bac8-837e-446b-9340-5fd94ba8be01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 00:18:58.167518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:58.167480 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf9bf0c-sc5xw" Oct 30 00:18:58.218844 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:58.218759 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0649729d-adb7-4ed6-90da-853e44f73236-kube-api-access-zgbch\" (UniqueName: \"kubernetes.io/projected/0649729d-adb7-4ed6-90da-853e44f73236-kube-api-access-zgbch\") pod \"0649729d-adb7-4ed6-90da-853e44f73236\" (UID: \"0649729d-adb7-4ed6-90da-853e44f73236\") " Oct 30 00:18:58.219549 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:58.219527 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0649729d-adb7-4ed6-90da-853e44f73236-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0649729d-adb7-4ed6-90da-853e44f73236-maas-api-xks-ca\") pod \"0649729d-adb7-4ed6-90da-853e44f73236\" (UID: \"0649729d-adb7-4ed6-90da-853e44f73236\") " Oct 30 00:18:58.220054 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:58.220035 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0649729d-adb7-4ed6-90da-853e44f73236-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0649729d-adb7-4ed6-90da-853e44f73236-maas-api-openshift-ca\") pod \"0649729d-adb7-4ed6-90da-853e44f73236\" (UID: \"0649729d-adb7-4ed6-90da-853e44f73236\") " Oct 30 00:18:58.220603 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:58.220500 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0649729d-adb7-4ed6-90da-853e44f73236-maas-api-openshift-ca" pod "0649729d-adb7-4ed6-90da-853e44f73236" (UID: "0649729d-adb7-4ed6-90da-853e44f73236"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:18:58.221329 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:58.221309 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0649729d-adb7-4ed6-90da-853e44f73236-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:18:58.258844 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:58.258807 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0649729d-adb7-4ed6-90da-853e44f73236-maas-api-xks-ca" pod "0649729d-adb7-4ed6-90da-853e44f73236" (UID: "0649729d-adb7-4ed6-90da-853e44f73236"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:18:58.260500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:58.260470 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0649729d-adb7-4ed6-90da-853e44f73236-kube-api-access-zgbch" pod "0649729d-adb7-4ed6-90da-853e44f73236" (UID: "0649729d-adb7-4ed6-90da-853e44f73236"). InnerVolumeSpecName "kube-api-access-zgbch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:18:58.322901 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:58.322859 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgbch\" (UniqueName: \"kubernetes.io/projected/0649729d-adb7-4ed6-90da-853e44f73236-kube-api-access-zgbch\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:18:58.323170 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:58.323149 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0649729d-adb7-4ed6-90da-853e44f73236-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:18:58.962331 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:58.962295 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf9bf0c-sc5xw" event={"ID":"0649729d-adb7-4ed6-90da-853e44f73236","Type":"ContainerDied","Data":"d277b00bc35b56a6904fe12c45de0a6516a79b19fbd65f1dbcc7fbeb8fa94034"} Oct 30 00:18:58.962331 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:58.962326 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf9bf0c-sc5xw" Oct 30 00:18:58.962547 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:58.962348 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d277b00bc35b56a6904fe12c45de0a6516a79b19fbd65f1dbcc7fbeb8fa94034" Oct 30 00:18:59.105932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:59.104251 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:18:59.106333 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:59.106125 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:18:59.122931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:59.117156 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:18:59.700619 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:59.700581 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f4006ca-gwj2w"] Oct 30 00:18:59.701198 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:59.701183 5001 cpu_manager.go:414] "containerMap: removing container" podUID="0649729d-adb7-4ed6-90da-853e44f73236" containerName="revoke-keys" Oct 30 00:18:59.701286 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:59.701199 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0649729d-adb7-4ed6-90da-853e44f73236" containerName="revoke-keys" Oct 30 00:18:59.701324 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:59.701316 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0649729d-adb7-4ed6-90da-853e44f73236" containerName="revoke-keys" Oct 30 00:18:59.704926 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:59.704894 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4006ca-gwj2w" Oct 30 00:18:59.718657 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:59.718636 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:18:59.718795 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:59.718693 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:18:59.718795 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:59.718762 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:18:59.728220 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:59.728196 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:18:59.733656 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:59.733632 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f4006ca-gwj2w"] Oct 30 00:18:59.746995 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:59.746968 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d3df50-bf97-4d1f-b622-1fbbf7b3734d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f4006ca-gwj2w\" (UID: \"44d3df50-bf97-4d1f-b622-1fbbf7b3734d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4006ca-gwj2w" Oct 30 00:18:59.747140 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:59.747120 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d3df50-bf97-4d1f-b622-1fbbf7b3734d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f4006ca-gwj2w\" (UID: \"44d3df50-bf97-4d1f-b622-1fbbf7b3734d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4006ca-gwj2w" Oct 30 00:18:59.747270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:59.747243 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5v6n\" (UniqueName: \"kubernetes.io/projected/44d3df50-bf97-4d1f-b622-1fbbf7b3734d-kube-api-access-k5v6n\") pod \"maas-api-revoke-sub-8f4006ca-gwj2w\" (UID: \"44d3df50-bf97-4d1f-b622-1fbbf7b3734d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4006ca-gwj2w" Oct 30 00:18:59.848877 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:59.848845 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5v6n\" (UniqueName: \"kubernetes.io/projected/44d3df50-bf97-4d1f-b622-1fbbf7b3734d-kube-api-access-k5v6n\") pod \"maas-api-revoke-sub-8f4006ca-gwj2w\" (UID: \"44d3df50-bf97-4d1f-b622-1fbbf7b3734d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4006ca-gwj2w" Oct 30 00:18:59.849068 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:59.848903 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d3df50-bf97-4d1f-b622-1fbbf7b3734d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f4006ca-gwj2w\" (UID: \"44d3df50-bf97-4d1f-b622-1fbbf7b3734d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4006ca-gwj2w" Oct 30 00:18:59.849068 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:59.849056 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d3df50-bf97-4d1f-b622-1fbbf7b3734d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f4006ca-gwj2w\" (UID: \"44d3df50-bf97-4d1f-b622-1fbbf7b3734d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4006ca-gwj2w" Oct 30 00:18:59.849647 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:59.849620 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d3df50-bf97-4d1f-b622-1fbbf7b3734d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f4006ca-gwj2w\" (UID: \"44d3df50-bf97-4d1f-b622-1fbbf7b3734d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4006ca-gwj2w" Oct 30 00:18:59.851272 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:59.851246 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d3df50-bf97-4d1f-b622-1fbbf7b3734d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f4006ca-gwj2w\" (UID: \"44d3df50-bf97-4d1f-b622-1fbbf7b3734d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4006ca-gwj2w" Oct 30 00:18:59.871765 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:18:59.871742 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5v6n\" (UniqueName: \"kubernetes.io/projected/44d3df50-bf97-4d1f-b622-1fbbf7b3734d-kube-api-access-k5v6n\") pod \"maas-api-revoke-sub-8f4006ca-gwj2w\" (UID: \"44d3df50-bf97-4d1f-b622-1fbbf7b3734d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4006ca-gwj2w" Oct 30 00:19:00.019174 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:00.019141 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4006ca-gwj2w" Oct 30 00:19:00.323080 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:00.323054 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f4006ca-gwj2w"] Oct 30 00:19:00.324971 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:19:00.324940 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d3df50_bf97_4d1f_b622_1fbbf7b3734d.slice/crio-886fb46738cbf3b7b8faab034c8e86374e6dc59aa43107acb50e1af9109d1ca5 WatchSource:0}: Error finding container 886fb46738cbf3b7b8faab034c8e86374e6dc59aa43107acb50e1af9109d1ca5: Status 404 returned error can't find the container with id 886fb46738cbf3b7b8faab034c8e86374e6dc59aa43107acb50e1af9109d1ca5 Oct 30 00:19:00.978159 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:00.978068 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4006ca-gwj2w" event={"ID":"44d3df50-bf97-4d1f-b622-1fbbf7b3734d","Type":"ContainerStarted","Data":"8306866e7c1dfd59aa7470424ce9271581eb1a209a96f3c8c0192ef19a4e246f"} Oct 30 00:19:00.978159 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:00.978127 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4006ca-gwj2w" event={"ID":"44d3df50-bf97-4d1f-b622-1fbbf7b3734d","Type":"ContainerStarted","Data":"886fb46738cbf3b7b8faab034c8e86374e6dc59aa43107acb50e1af9109d1ca5"} Oct 30 00:19:01.016732 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:01.016675 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4006ca-gwj2w" podStartSLOduration=2.016661262 podStartE2EDuration="2.016661262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:19:01.012447747 +0000 UTC m=+856.217948930" watchObservedRunningTime="2026-10-30 00:19:01.016661262 +0000 UTC m=+856.222162442" Oct 30 00:19:01.987999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:01.987965 5001 generic.go:356] "Generic (PLEG): container finished" podID="44d3df50-bf97-4d1f-b622-1fbbf7b3734d" containerID="8306866e7c1dfd59aa7470424ce9271581eb1a209a96f3c8c0192ef19a4e246f" exitCode=0 Oct 30 00:19:01.988377 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:01.988051 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4006ca-gwj2w" event={"ID":"44d3df50-bf97-4d1f-b622-1fbbf7b3734d","Type":"ContainerDied","Data":"8306866e7c1dfd59aa7470424ce9271581eb1a209a96f3c8c0192ef19a4e246f"} Oct 30 00:19:03.156619 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:03.156592 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4006ca-gwj2w" Oct 30 00:19:03.195923 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:03.195870 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44d3df50-bf97-4d1f-b622-1fbbf7b3734d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d3df50-bf97-4d1f-b622-1fbbf7b3734d-maas-api-xks-ca\") pod \"44d3df50-bf97-4d1f-b622-1fbbf7b3734d\" (UID: \"44d3df50-bf97-4d1f-b622-1fbbf7b3734d\") " Oct 30 00:19:03.195923 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:03.195997 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44d3df50-bf97-4d1f-b622-1fbbf7b3734d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d3df50-bf97-4d1f-b622-1fbbf7b3734d-maas-api-openshift-ca\") pod \"44d3df50-bf97-4d1f-b622-1fbbf7b3734d\" (UID: \"44d3df50-bf97-4d1f-b622-1fbbf7b3734d\") " Oct 30 00:19:03.195923 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:03.196028 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44d3df50-bf97-4d1f-b622-1fbbf7b3734d-kube-api-access-k5v6n\" (UniqueName: \"kubernetes.io/projected/44d3df50-bf97-4d1f-b622-1fbbf7b3734d-kube-api-access-k5v6n\") pod \"44d3df50-bf97-4d1f-b622-1fbbf7b3734d\" (UID: \"44d3df50-bf97-4d1f-b622-1fbbf7b3734d\") " Oct 30 00:19:03.196768 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:03.196717 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44d3df50-bf97-4d1f-b622-1fbbf7b3734d-maas-api-openshift-ca" pod "44d3df50-bf97-4d1f-b622-1fbbf7b3734d" (UID: "44d3df50-bf97-4d1f-b622-1fbbf7b3734d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:19:03.198886 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:03.198863 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44d3df50-bf97-4d1f-b622-1fbbf7b3734d-maas-api-xks-ca" pod "44d3df50-bf97-4d1f-b622-1fbbf7b3734d" (UID: "44d3df50-bf97-4d1f-b622-1fbbf7b3734d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:19:03.200203 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:03.200120 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d3df50-bf97-4d1f-b622-1fbbf7b3734d-kube-api-access-k5v6n" pod "44d3df50-bf97-4d1f-b622-1fbbf7b3734d" (UID: "44d3df50-bf97-4d1f-b622-1fbbf7b3734d"). InnerVolumeSpecName "kube-api-access-k5v6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:19:03.298263 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:03.298228 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d3df50-bf97-4d1f-b622-1fbbf7b3734d-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:19:03.298263 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:03.298260 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d3df50-bf97-4d1f-b622-1fbbf7b3734d-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:19:03.298466 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:03.298275 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5v6n\" (UniqueName: \"kubernetes.io/projected/44d3df50-bf97-4d1f-b622-1fbbf7b3734d-kube-api-access-k5v6n\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:19:04.004353 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.004327 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4006ca-gwj2w" Oct 30 00:19:04.004579 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.004324 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4006ca-gwj2w" event={"ID":"44d3df50-bf97-4d1f-b622-1fbbf7b3734d","Type":"ContainerDied","Data":"886fb46738cbf3b7b8faab034c8e86374e6dc59aa43107acb50e1af9109d1ca5"} Oct 30 00:19:04.004579 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.004452 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="886fb46738cbf3b7b8faab034c8e86374e6dc59aa43107acb50e1af9109d1ca5" Oct 30 00:19:04.064698 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:19:04.064667 5001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d3df50_bf97_4d1f_b622_1fbbf7b3734d.slice\": RecentStats: unable to find data in memory cache]" Oct 30 00:19:04.172851 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.172801 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc"] Oct 30 00:19:04.174682 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.174462 5001 cpu_manager.go:414] "containerMap: removing container" podUID="44d3df50-bf97-4d1f-b622-1fbbf7b3734d" containerName="revoke-keys" Oct 30 00:19:04.174682 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.174503 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44d3df50-bf97-4d1f-b622-1fbbf7b3734d" containerName="revoke-keys" Oct 30 00:19:04.174682 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.174675 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="44d3df50-bf97-4d1f-b622-1fbbf7b3734d" containerName="revoke-keys" Oct 30 00:19:04.180532 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.180506 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:19:04.202774 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.202743 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc"] Oct 30 00:19:04.209255 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.209224 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-workload-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:19:04.209408 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.209344 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73a204ce-06bf-46b8-a99e-a26007c69e8b-istio-podinfo\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:19:04.209458 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.209408 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9vp9\" (UniqueName: \"kubernetes.io/projected/73a204ce-06bf-46b8-a99e-a26007c69e8b-kube-api-access-l9vp9\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:19:04.209458 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.209440 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-workload-certs\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:19:04.209532 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.209518 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73a204ce-06bf-46b8-a99e-a26007c69e8b-istio-token\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:19:04.209572 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.209557 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-credential-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:19:04.209722 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.209702 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-istio-envoy\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:19:04.209806 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.209736 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-istio-data\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:19:04.209868 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.209816 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73a204ce-06bf-46b8-a99e-a26007c69e8b-istiod-ca-cert\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:19:04.311243 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.311208 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73a204ce-06bf-46b8-a99e-a26007c69e8b-istiod-ca-cert\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:19:04.311417 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.311266 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-workload-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:19:04.311417 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.311299 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73a204ce-06bf-46b8-a99e-a26007c69e8b-istio-podinfo\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:19:04.311417 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.311325 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9vp9\" (UniqueName: \"kubernetes.io/projected/73a204ce-06bf-46b8-a99e-a26007c69e8b-kube-api-access-l9vp9\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:19:04.311417 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.311345 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-workload-certs\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:19:04.311417 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.311388 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73a204ce-06bf-46b8-a99e-a26007c69e8b-istio-token\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:19:04.311417 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.311416 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-credential-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:19:04.311746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.311470 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-istio-envoy\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:19:04.311746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.311495 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-istio-data\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:19:04.311847 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.311775 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-workload-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:19:04.311847 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.311814 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-workload-certs\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:19:04.312003 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.311877 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-credential-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:19:04.312120 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.312101 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-istio-data\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:19:04.312159 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.312100 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73a204ce-06bf-46b8-a99e-a26007c69e8b-istiod-ca-cert\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:19:04.313808 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.313784 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-istio-envoy\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:19:04.313975 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.313956 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73a204ce-06bf-46b8-a99e-a26007c69e8b-istio-podinfo\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:19:04.332157 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.332130 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73a204ce-06bf-46b8-a99e-a26007c69e8b-istio-token\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:19:04.332624 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.332597 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9vp9\" (UniqueName: \"kubernetes.io/projected/73a204ce-06bf-46b8-a99e-a26007c69e8b-kube-api-access-l9vp9\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:19:04.509757 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.509724 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:19:04.832961 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.831749 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc"] Oct 30 00:19:04.833162 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:19:04.833135 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a204ce_06bf_46b8_a99e_a26007c69e8b.slice/crio-bd99b24c239006c0c96169fc74a544d2ca0ef2b4f2fc0ff320c6f4200281065f WatchSource:0}: Error finding container bd99b24c239006c0c96169fc74a544d2ca0ef2b4f2fc0ff320c6f4200281065f: Status 404 returned error can't find the container with id bd99b24c239006c0c96169fc74a544d2ca0ef2b4f2fc0ff320c6f4200281065f Oct 30 00:19:04.836431 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.836393 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:19:04.836500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.836477 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:19:04.836551 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:04.836520 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:19:05.078868 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:05.078828 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" event={"ID":"73a204ce-06bf-46b8-a99e-a26007c69e8b","Type":"ContainerStarted","Data":"67241d3224c936ed20a6649fe2c91a692ea423853d15354fa25174a9db40d090"} Oct 30 00:19:05.079168 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:05.079146 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" event={"ID":"73a204ce-06bf-46b8-a99e-a26007c69e8b","Type":"ContainerStarted","Data":"bd99b24c239006c0c96169fc74a544d2ca0ef2b4f2fc0ff320c6f4200281065f"} Oct 30 00:19:05.135631 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:05.135420 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" podStartSLOduration=1.135399807 podStartE2EDuration="1.135399807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:19:05.12832091 +0000 UTC m=+860.333822119" watchObservedRunningTime="2026-10-30 00:19:05.135399807 +0000 UTC m=+860.340900991" Oct 30 00:19:05.514114 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:05.514067 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:19:05.514643 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:05.514253 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 00:19:05.514643 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:05.514296 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" podUID="73a204ce-06bf-46b8-a99e-a26007c69e8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 00:19:06.528935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:06.522264 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 00:19:06.528935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:06.522324 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" podUID="73a204ce-06bf-46b8-a99e-a26007c69e8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 00:19:07.519938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:07.512243 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 00:19:07.519938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:07.512299 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" podUID="73a204ce-06bf-46b8-a99e-a26007c69e8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 00:19:08.520079 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:08.512234 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 00:19:08.520079 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:08.512296 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" podUID="73a204ce-06bf-46b8-a99e-a26007c69e8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 00:19:08.639638 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:08.639598 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3675dc8c-h57sw"] Oct 30 00:19:08.645156 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:08.645130 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675dc8c-h57sw" Oct 30 00:19:08.664201 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:08.664172 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3675dc8c-h57sw"] Oct 30 00:19:08.667858 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:08.667828 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:19:08.668015 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:08.667974 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:19:08.668095 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:08.668046 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:19:08.668170 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:08.668134 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:19:08.679958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:08.679922 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2123cf26-a896-471e-9d46-7916c242407c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3675dc8c-h57sw\" (UID: \"2123cf26-a896-471e-9d46-7916c242407c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675dc8c-h57sw" Oct 30 00:19:08.680103 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:08.680024 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzqbz\" (UniqueName: \"kubernetes.io/projected/2123cf26-a896-471e-9d46-7916c242407c-kube-api-access-vzqbz\") pod \"maas-api-revoke-sub-3675dc8c-h57sw\" (UID: \"2123cf26-a896-471e-9d46-7916c242407c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675dc8c-h57sw" Oct 30 00:19:08.680489 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:08.680208 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2123cf26-a896-471e-9d46-7916c242407c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3675dc8c-h57sw\" (UID: \"2123cf26-a896-471e-9d46-7916c242407c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675dc8c-h57sw" Oct 30 00:19:08.782143 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:08.782045 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2123cf26-a896-471e-9d46-7916c242407c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3675dc8c-h57sw\" (UID: \"2123cf26-a896-471e-9d46-7916c242407c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675dc8c-h57sw" Oct 30 00:19:08.782298 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:08.782241 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzqbz\" (UniqueName: \"kubernetes.io/projected/2123cf26-a896-471e-9d46-7916c242407c-kube-api-access-vzqbz\") pod \"maas-api-revoke-sub-3675dc8c-h57sw\" (UID: \"2123cf26-a896-471e-9d46-7916c242407c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675dc8c-h57sw" Oct 30 00:19:08.782395 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:08.782381 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2123cf26-a896-471e-9d46-7916c242407c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3675dc8c-h57sw\" (UID: \"2123cf26-a896-471e-9d46-7916c242407c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675dc8c-h57sw" Oct 30 00:19:08.783189 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:08.783167 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2123cf26-a896-471e-9d46-7916c242407c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3675dc8c-h57sw\" (UID: \"2123cf26-a896-471e-9d46-7916c242407c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675dc8c-h57sw" Oct 30 00:19:08.784448 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:08.784427 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2123cf26-a896-471e-9d46-7916c242407c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3675dc8c-h57sw\" (UID: \"2123cf26-a896-471e-9d46-7916c242407c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675dc8c-h57sw" Oct 30 00:19:08.802830 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:08.802805 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzqbz\" (UniqueName: \"kubernetes.io/projected/2123cf26-a896-471e-9d46-7916c242407c-kube-api-access-vzqbz\") pod \"maas-api-revoke-sub-3675dc8c-h57sw\" (UID: \"2123cf26-a896-471e-9d46-7916c242407c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675dc8c-h57sw" Oct 30 00:19:08.960355 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:08.960327 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675dc8c-h57sw" Oct 30 00:19:09.292722 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:09.292696 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3675dc8c-h57sw"] Oct 30 00:19:09.294218 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:19:09.294194 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2123cf26_a896_471e_9d46_7916c242407c.slice/crio-2201f518cc8ec672be8fccdd25df7de0b3a45510b48468e42d9977562baed1d8 WatchSource:0}: Error finding container 2201f518cc8ec672be8fccdd25df7de0b3a45510b48468e42d9977562baed1d8: Status 404 returned error can't find the container with id 2201f518cc8ec672be8fccdd25df7de0b3a45510b48468e42d9977562baed1d8 Oct 30 00:19:09.525314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:09.525279 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 00:19:09.525880 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:09.525846 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" podUID="73a204ce-06bf-46b8-a99e-a26007c69e8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 00:19:10.151684 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:10.151645 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675dc8c-h57sw" event={"ID":"2123cf26-a896-471e-9d46-7916c242407c","Type":"ContainerStarted","Data":"7df666dee96e08d14d85fb3df3b65106261c97198e55703225d211e91b144aa8"} Oct 30 00:19:10.151877 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:10.151708 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675dc8c-h57sw" event={"ID":"2123cf26-a896-471e-9d46-7916c242407c","Type":"ContainerStarted","Data":"2201f518cc8ec672be8fccdd25df7de0b3a45510b48468e42d9977562baed1d8"} Oct 30 00:19:10.189606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:10.189549 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675dc8c-h57sw" podStartSLOduration=2.189533254 podStartE2EDuration="2.189533254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:19:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:19:10.187133353 +0000 UTC m=+865.392634531" watchObservedRunningTime="2026-10-30 00:19:10.189533254 +0000 UTC m=+865.395034482" Oct 30 00:19:10.529936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:10.512209 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 00:19:10.529936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:10.512266 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" podUID="73a204ce-06bf-46b8-a99e-a26007c69e8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 00:19:11.161217 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:11.161173 5001 generic.go:356] "Generic (PLEG): container finished" podID="2123cf26-a896-471e-9d46-7916c242407c" containerID="7df666dee96e08d14d85fb3df3b65106261c97198e55703225d211e91b144aa8" exitCode=0 Oct 30 00:19:11.161451 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:11.161252 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675dc8c-h57sw" event={"ID":"2123cf26-a896-471e-9d46-7916c242407c","Type":"ContainerDied","Data":"7df666dee96e08d14d85fb3df3b65106261c97198e55703225d211e91b144aa8"} Oct 30 00:19:11.510242 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:11.510212 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 00:19:11.510388 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:11.510277 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" podUID="73a204ce-06bf-46b8-a99e-a26007c69e8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 00:19:12.360760 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:12.360736 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675dc8c-h57sw" Oct 30 00:19:12.444607 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:12.444518 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2123cf26-a896-471e-9d46-7916c242407c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2123cf26-a896-471e-9d46-7916c242407c-maas-api-openshift-ca\") pod \"2123cf26-a896-471e-9d46-7916c242407c\" (UID: \"2123cf26-a896-471e-9d46-7916c242407c\") " Oct 30 00:19:12.444607 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:12.444601 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2123cf26-a896-471e-9d46-7916c242407c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2123cf26-a896-471e-9d46-7916c242407c-maas-api-xks-ca\") pod \"2123cf26-a896-471e-9d46-7916c242407c\" (UID: \"2123cf26-a896-471e-9d46-7916c242407c\") " Oct 30 00:19:12.444806 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:12.444629 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2123cf26-a896-471e-9d46-7916c242407c-kube-api-access-vzqbz\" (UniqueName: \"kubernetes.io/projected/2123cf26-a896-471e-9d46-7916c242407c-kube-api-access-vzqbz\") pod \"2123cf26-a896-471e-9d46-7916c242407c\" (UID: \"2123cf26-a896-471e-9d46-7916c242407c\") " Oct 30 00:19:12.445022 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:12.444990 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2123cf26-a896-471e-9d46-7916c242407c-maas-api-openshift-ca" pod "2123cf26-a896-471e-9d46-7916c242407c" (UID: "2123cf26-a896-471e-9d46-7916c242407c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:19:12.445219 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:12.445199 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2123cf26-a896-471e-9d46-7916c242407c-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:19:12.446797 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:12.446767 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2123cf26-a896-471e-9d46-7916c242407c-kube-api-access-vzqbz" pod "2123cf26-a896-471e-9d46-7916c242407c" (UID: "2123cf26-a896-471e-9d46-7916c242407c"). InnerVolumeSpecName "kube-api-access-vzqbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:19:12.446936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:12.446794 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2123cf26-a896-471e-9d46-7916c242407c-maas-api-xks-ca" pod "2123cf26-a896-471e-9d46-7916c242407c" (UID: "2123cf26-a896-471e-9d46-7916c242407c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:19:12.515234 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:12.515204 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:19:12.515680 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:12.515662 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:19:12.516810 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:12.516792 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:19:12.546624 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:12.546600 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2123cf26-a896-471e-9d46-7916c242407c-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:19:12.546624 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:12.546622 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzqbz\" (UniqueName: \"kubernetes.io/projected/2123cf26-a896-471e-9d46-7916c242407c-kube-api-access-vzqbz\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:19:13.180015 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:13.179983 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675dc8c-h57sw" Oct 30 00:19:13.180015 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:13.180006 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675dc8c-h57sw" event={"ID":"2123cf26-a896-471e-9d46-7916c242407c","Type":"ContainerDied","Data":"2201f518cc8ec672be8fccdd25df7de0b3a45510b48468e42d9977562baed1d8"} Oct 30 00:19:13.180252 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:13.180055 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2201f518cc8ec672be8fccdd25df7de0b3a45510b48468e42d9977562baed1d8" Oct 30 00:19:16.454883 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.454846 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m"] Oct 30 00:19:16.455429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.455415 5001 cpu_manager.go:414] "containerMap: removing container" podUID="2123cf26-a896-471e-9d46-7916c242407c" containerName="revoke-keys" Oct 30 00:19:16.455468 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.455430 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2123cf26-a896-471e-9d46-7916c242407c" containerName="revoke-keys" Oct 30 00:19:16.455546 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.455538 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2123cf26-a896-471e-9d46-7916c242407c" containerName="revoke-keys" Oct 30 00:19:16.459354 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.459328 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:19:16.487482 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.487449 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m"] Oct 30 00:19:16.548844 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.548803 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:19:16.549036 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.548868 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:19:16.549036 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.548974 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:19:16.549036 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.549008 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:19:16.549184 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.549041 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpsn5\" (UniqueName: \"kubernetes.io/projected/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-kube-api-access-bpsn5\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:19:16.549184 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.549087 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:19:16.549283 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.549194 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:19:16.549283 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.549254 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:19:16.549386 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.549325 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:19:16.650871 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.650837 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:19:16.651085 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.650885 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:19:16.651085 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.650936 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpsn5\" (UniqueName: \"kubernetes.io/projected/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-kube-api-access-bpsn5\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:19:16.651085 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.650975 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:19:16.651085 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.651019 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:19:16.651085 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.651065 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:19:16.651339 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.651139 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:19:16.651339 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.651294 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:19:16.651339 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.651324 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:19:16.651482 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.651363 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:19:16.651482 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.651419 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:19:16.651482 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.651447 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:19:16.651669 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.651609 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:19:16.652044 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.652022 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:19:16.653596 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.653574 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:19:16.653724 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.653705 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:19:16.671559 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.671533 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:19:16.671949 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.671925 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpsn5\" (UniqueName: \"kubernetes.io/projected/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-kube-api-access-bpsn5\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:19:16.773325 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:16.773292 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:19:17.091822 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:17.091796 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m"] Oct 30 00:19:17.093458 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:19:17.093430 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38fbc17b_0fb6_46c8_83af_53b83fa2bcb1.slice/crio-e694ebafde43c1e0180b70777d44d56a2bc337709658cc8b509724f068ecf20f WatchSource:0}: Error finding container e694ebafde43c1e0180b70777d44d56a2bc337709658cc8b509724f068ecf20f: Status 404 returned error can't find the container with id e694ebafde43c1e0180b70777d44d56a2bc337709658cc8b509724f068ecf20f Oct 30 00:19:17.095792 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:17.095758 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:19:17.095880 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:17.095824 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:19:17.095880 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:17.095871 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:19:17.295366 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:17.295326 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" event={"ID":"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1","Type":"ContainerStarted","Data":"add4d613b8644e6634338b28efdc601d2fa4cad732d30520b5434a4cdd351bbe"} Oct 30 00:19:17.295539 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:17.295394 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" event={"ID":"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1","Type":"ContainerStarted","Data":"e694ebafde43c1e0180b70777d44d56a2bc337709658cc8b509724f068ecf20f"} Oct 30 00:19:17.329257 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:19:17.329197 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:19:17.338260 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:17.338211 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" podStartSLOduration=1.33819675 podStartE2EDuration="1.33819675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:19:17.3353334 +0000 UTC m=+872.540834605" watchObservedRunningTime="2026-10-30 00:19:17.33819675 +0000 UTC m=+872.543697930" Oct 30 00:19:17.779784 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:17.776771 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:19:18.775282 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:18.775244 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 00:19:18.775457 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:18.775298 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" podUID="38fbc17b-0fb6-46c8-83af-53b83fa2bcb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 00:19:19.084064 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:19.080984 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56e7f57d-p8g94"] Oct 30 00:19:19.089941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:19.089698 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e7f57d-p8g94" Oct 30 00:19:19.096933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:19.095893 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d42bb9aa-fb91-462a-a01f-01c821cc2bb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56e7f57d-p8g94\" (UID: \"d42bb9aa-fb91-462a-a01f-01c821cc2bb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e7f57d-p8g94" Oct 30 00:19:19.096933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:19.095959 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d42bb9aa-fb91-462a-a01f-01c821cc2bb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56e7f57d-p8g94\" (UID: \"d42bb9aa-fb91-462a-a01f-01c821cc2bb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e7f57d-p8g94" Oct 30 00:19:19.096933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:19.095992 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd55c\" (UniqueName: \"kubernetes.io/projected/d42bb9aa-fb91-462a-a01f-01c821cc2bb6-kube-api-access-hd55c\") pod \"maas-api-revoke-sub-56e7f57d-p8g94\" (UID: \"d42bb9aa-fb91-462a-a01f-01c821cc2bb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e7f57d-p8g94" Oct 30 00:19:19.120930 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:19.112284 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:19:19.120930 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:19.112352 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:19:19.140939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:19.122245 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:19:19.140939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:19.122373 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:19:19.140939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:19.133278 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56e7f57d-p8g94"] Oct 30 00:19:19.200937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:19.197139 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d42bb9aa-fb91-462a-a01f-01c821cc2bb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56e7f57d-p8g94\" (UID: \"d42bb9aa-fb91-462a-a01f-01c821cc2bb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e7f57d-p8g94" Oct 30 00:19:19.200937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:19.197214 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d42bb9aa-fb91-462a-a01f-01c821cc2bb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56e7f57d-p8g94\" (UID: \"d42bb9aa-fb91-462a-a01f-01c821cc2bb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e7f57d-p8g94" Oct 30 00:19:19.200937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:19.197256 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd55c\" (UniqueName: \"kubernetes.io/projected/d42bb9aa-fb91-462a-a01f-01c821cc2bb6-kube-api-access-hd55c\") pod \"maas-api-revoke-sub-56e7f57d-p8g94\" (UID: \"d42bb9aa-fb91-462a-a01f-01c821cc2bb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e7f57d-p8g94" Oct 30 00:19:19.200937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:19.199600 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d42bb9aa-fb91-462a-a01f-01c821cc2bb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56e7f57d-p8g94\" (UID: \"d42bb9aa-fb91-462a-a01f-01c821cc2bb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e7f57d-p8g94" Oct 30 00:19:19.218942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:19.202765 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d42bb9aa-fb91-462a-a01f-01c821cc2bb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56e7f57d-p8g94\" (UID: \"d42bb9aa-fb91-462a-a01f-01c821cc2bb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e7f57d-p8g94" Oct 30 00:19:19.273939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:19.273695 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd55c\" (UniqueName: \"kubernetes.io/projected/d42bb9aa-fb91-462a-a01f-01c821cc2bb6-kube-api-access-hd55c\") pod \"maas-api-revoke-sub-56e7f57d-p8g94\" (UID: \"d42bb9aa-fb91-462a-a01f-01c821cc2bb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e7f57d-p8g94" Oct 30 00:19:19.492764 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:19.492673 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e7f57d-p8g94" Oct 30 00:19:19.774807 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:19.774329 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 00:19:19.774807 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:19.774404 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" podUID="38fbc17b-0fb6-46c8-83af-53b83fa2bcb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 00:19:20.507080 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:20.506975 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:19:20.649516 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:20.643793 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56e7f57d-p8g94"] Oct 30 00:19:20.663143 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:20.663100 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:20.701576 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:20.701528 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:19:20.774682 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:20.774267 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 00:19:20.774682 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:20.774322 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" podUID="38fbc17b-0fb6-46c8-83af-53b83fa2bcb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 00:19:20.924937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:20.924007 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:21.092934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:21.088039 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:21.384269 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:21.383240 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 00:19:21.479128 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:21.479066 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:21.548633 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:21.548566 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:19:21.660996 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:21.629212 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:19:21.755295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:21.690377 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:19:21.781942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:21.757747 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:19:21.781942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:21.709671 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:19:21.806938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:21.784064 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:19:21.809371 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:21.809332 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:21.809676 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:21.809592 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 00:19:21.809806 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:21.809785 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" podUID="38fbc17b-0fb6-46c8-83af-53b83fa2bcb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 00:19:22.062946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:22.061007 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:22.062946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:22.061362 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:19:22.088992 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:22.088952 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 00:19:22.120140 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:22.111188 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e7f57d-p8g94" event={"ID":"d42bb9aa-fb91-462a-a01f-01c821cc2bb6","Type":"ContainerStarted","Data":"b726d60724c98a99053d6dfcedf5a7304e311c70a9240f3e70945dc3684d41d8"} Oct 30 00:19:22.120455 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:22.120434 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:19:22.121032 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:22.121011 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e7f57d-p8g94" event={"ID":"d42bb9aa-fb91-462a-a01f-01c821cc2bb6","Type":"ContainerStarted","Data":"6b3043349ad12c2995eaf95c9bb324738c13894d3e7cbaad131115fa91de753a"} Oct 30 00:19:22.401239 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:22.401137 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:22.407937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:22.403232 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:19:22.474935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:22.460342 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e7f57d-p8g94" podStartSLOduration=3.460320004 podStartE2EDuration="3.460320004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:19:22.198339739 +0000 UTC m=+877.403840921" watchObservedRunningTime="2026-10-30 00:19:22.460320004 +0000 UTC m=+877.665821208" Oct 30 00:19:22.484375 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:22.483009 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:22.506049 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:22.484630 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:19:22.631147 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:22.629993 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:22.769994 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:22.758020 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:19:22.769994 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:22.763322 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:22.805937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:22.805080 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 00:19:22.805937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:22.805146 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" podUID="38fbc17b-0fb6-46c8-83af-53b83fa2bcb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 00:19:23.069059 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:23.068964 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:23.088141 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:23.088111 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:19:23.140937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:23.132366 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 00:19:23.293935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:23.293213 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:19:23.293935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:23.293283 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" podUID="922f38ba-e017-4005-9841-aa0fd46b1d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:23.301776 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:23.301740 5001 generic.go:356] "Generic (PLEG): container finished" podID="d42bb9aa-fb91-462a-a01f-01c821cc2bb6" containerID="b726d60724c98a99053d6dfcedf5a7304e311c70a9240f3e70945dc3684d41d8" exitCode=0 Oct 30 00:19:23.314693 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:23.312372 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e7f57d-p8g94" event={"ID":"d42bb9aa-fb91-462a-a01f-01c821cc2bb6","Type":"ContainerDied","Data":"b726d60724c98a99053d6dfcedf5a7304e311c70a9240f3e70945dc3684d41d8"} Oct 30 00:19:23.416328 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:23.414931 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:23.491937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:23.483512 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:23.782202 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:23.782162 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:23.786059 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:23.786026 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:19:23.787885 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:23.786484 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 00:19:23.788097 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:23.788073 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" podUID="38fbc17b-0fb6-46c8-83af-53b83fa2bcb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 00:19:24.138334 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:24.138238 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:24.138808 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:24.138782 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:24.140654 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:24.140625 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:24.179215 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:24.179174 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:24.209787 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:24.209755 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:19:24.305047 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:24.305013 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:19:24.506980 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:24.486161 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:24.812936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:24.801047 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:24.812936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:24.802395 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 00:19:24.812936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:24.802439 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" podUID="38fbc17b-0fb6-46c8-83af-53b83fa2bcb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 00:19:25.148735 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:25.148613 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:25.151560 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:25.151523 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:19:25.252902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:25.239309 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:25.456123 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:25.266973 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:25.620078 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:25.620036 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:25.647398 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:25.647354 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:19:25.663732 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:25.650109 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f31177a-9fv92"] Oct 30 00:19:25.663732 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:25.661240 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f31177a-9fv92"] Oct 30 00:19:25.663732 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:25.661378 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f31177a-9fv92" Oct 30 00:19:25.695906 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:25.695869 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:19:25.775345 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:25.770309 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:19:25.791707 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:25.791677 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzww9\" (UniqueName: \"kubernetes.io/projected/1e1a4c0b-7bda-4ada-8221-bd444286bd94-kube-api-access-hzww9\") pod \"maas-api-revoke-sub-2f31177a-9fv92\" (UID: \"1e1a4c0b-7bda-4ada-8221-bd444286bd94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f31177a-9fv92" Oct 30 00:19:25.792060 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:25.792037 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e1a4c0b-7bda-4ada-8221-bd444286bd94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f31177a-9fv92\" (UID: \"1e1a4c0b-7bda-4ada-8221-bd444286bd94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f31177a-9fv92" Oct 30 00:19:25.792294 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:25.792275 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e1a4c0b-7bda-4ada-8221-bd444286bd94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f31177a-9fv92\" (UID: \"1e1a4c0b-7bda-4ada-8221-bd444286bd94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f31177a-9fv92" Oct 30 00:19:25.795056 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:25.795035 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 00:19:25.795202 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:25.795179 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" podUID="38fbc17b-0fb6-46c8-83af-53b83fa2bcb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 00:19:25.797599 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:25.797428 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:19:25.807289 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:25.803002 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:19:25.846231 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:25.846203 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:19:25.936722 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:25.936691 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzww9\" (UniqueName: \"kubernetes.io/projected/1e1a4c0b-7bda-4ada-8221-bd444286bd94-kube-api-access-hzww9\") pod \"maas-api-revoke-sub-2f31177a-9fv92\" (UID: \"1e1a4c0b-7bda-4ada-8221-bd444286bd94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f31177a-9fv92" Oct 30 00:19:25.950657 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:25.950627 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e1a4c0b-7bda-4ada-8221-bd444286bd94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f31177a-9fv92\" (UID: \"1e1a4c0b-7bda-4ada-8221-bd444286bd94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f31177a-9fv92" Oct 30 00:19:25.950902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:25.950881 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e1a4c0b-7bda-4ada-8221-bd444286bd94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f31177a-9fv92\" (UID: \"1e1a4c0b-7bda-4ada-8221-bd444286bd94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f31177a-9fv92" Oct 30 00:19:25.952433 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:25.952126 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e1a4c0b-7bda-4ada-8221-bd444286bd94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f31177a-9fv92\" (UID: \"1e1a4c0b-7bda-4ada-8221-bd444286bd94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f31177a-9fv92" Oct 30 00:19:25.953648 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:25.953312 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e7f57d-p8g94" Oct 30 00:19:25.960650 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:25.960627 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzww9\" (UniqueName: \"kubernetes.io/projected/1e1a4c0b-7bda-4ada-8221-bd444286bd94-kube-api-access-hzww9\") pod \"maas-api-revoke-sub-2f31177a-9fv92\" (UID: \"1e1a4c0b-7bda-4ada-8221-bd444286bd94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f31177a-9fv92" Oct 30 00:19:25.973455 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:25.973402 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e1a4c0b-7bda-4ada-8221-bd444286bd94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f31177a-9fv92\" (UID: \"1e1a4c0b-7bda-4ada-8221-bd444286bd94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f31177a-9fv92" Oct 30 00:19:26.029453 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:26.029153 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f31177a-9fv92" Oct 30 00:19:26.058162 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:26.058133 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d42bb9aa-fb91-462a-a01f-01c821cc2bb6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d42bb9aa-fb91-462a-a01f-01c821cc2bb6-maas-api-openshift-ca\") pod \"d42bb9aa-fb91-462a-a01f-01c821cc2bb6\" (UID: \"d42bb9aa-fb91-462a-a01f-01c821cc2bb6\") " Oct 30 00:19:26.058516 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:26.058492 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d42bb9aa-fb91-462a-a01f-01c821cc2bb6-kube-api-access-hd55c\" (UniqueName: \"kubernetes.io/projected/d42bb9aa-fb91-462a-a01f-01c821cc2bb6-kube-api-access-hd55c\") pod \"d42bb9aa-fb91-462a-a01f-01c821cc2bb6\" (UID: \"d42bb9aa-fb91-462a-a01f-01c821cc2bb6\") " Oct 30 00:19:26.059141 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:26.058781 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d42bb9aa-fb91-462a-a01f-01c821cc2bb6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d42bb9aa-fb91-462a-a01f-01c821cc2bb6-maas-api-xks-ca\") pod \"d42bb9aa-fb91-462a-a01f-01c821cc2bb6\" (UID: \"d42bb9aa-fb91-462a-a01f-01c821cc2bb6\") " Oct 30 00:19:26.060124 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:26.060099 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d42bb9aa-fb91-462a-a01f-01c821cc2bb6-maas-api-openshift-ca" pod "d42bb9aa-fb91-462a-a01f-01c821cc2bb6" (UID: "d42bb9aa-fb91-462a-a01f-01c821cc2bb6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:19:26.063226 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:26.063201 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d42bb9aa-fb91-462a-a01f-01c821cc2bb6-kube-api-access-hd55c" pod "d42bb9aa-fb91-462a-a01f-01c821cc2bb6" (UID: "d42bb9aa-fb91-462a-a01f-01c821cc2bb6"). InnerVolumeSpecName "kube-api-access-hd55c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:19:26.067109 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:26.067072 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d42bb9aa-fb91-462a-a01f-01c821cc2bb6-maas-api-xks-ca" pod "d42bb9aa-fb91-462a-a01f-01c821cc2bb6" (UID: "d42bb9aa-fb91-462a-a01f-01c821cc2bb6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:19:26.068470 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:26.067489 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:19:26.161291 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:26.161252 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d42bb9aa-fb91-462a-a01f-01c821cc2bb6-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:19:26.161542 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:26.161523 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d42bb9aa-fb91-462a-a01f-01c821cc2bb6-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:19:26.161678 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:26.161662 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd55c\" (UniqueName: \"kubernetes.io/projected/d42bb9aa-fb91-462a-a01f-01c821cc2bb6-kube-api-access-hd55c\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:19:26.448045 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:26.448018 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f31177a-9fv92"] Oct 30 00:19:26.449396 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:19:26.449369 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e1a4c0b_7bda_4ada_8221_bd444286bd94.slice/crio-4d600cf5e192b8eb70234c5598bfe1fe0e427d9ceda4b712e1e199d9b5996918 WatchSource:0}: Error finding container 4d600cf5e192b8eb70234c5598bfe1fe0e427d9ceda4b712e1e199d9b5996918: Status 404 returned error can't find the container with id 4d600cf5e192b8eb70234c5598bfe1fe0e427d9ceda4b712e1e199d9b5996918 Oct 30 00:19:26.685544 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:26.685516 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e7f57d-p8g94" Oct 30 00:19:26.685544 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:26.685539 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e7f57d-p8g94" event={"ID":"d42bb9aa-fb91-462a-a01f-01c821cc2bb6","Type":"ContainerDied","Data":"6b3043349ad12c2995eaf95c9bb324738c13894d3e7cbaad131115fa91de753a"} Oct 30 00:19:26.685747 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:26.685584 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b3043349ad12c2995eaf95c9bb324738c13894d3e7cbaad131115fa91de753a" Oct 30 00:19:26.686945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:26.686907 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f31177a-9fv92" event={"ID":"1e1a4c0b-7bda-4ada-8221-bd444286bd94","Type":"ContainerStarted","Data":"4d600cf5e192b8eb70234c5598bfe1fe0e427d9ceda4b712e1e199d9b5996918"} Oct 30 00:19:26.774567 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:26.774523 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 00:19:26.774715 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:26.774592 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" podUID="38fbc17b-0fb6-46c8-83af-53b83fa2bcb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 00:19:27.707883 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:27.707784 5001 generic.go:356] "Generic (PLEG): container finished" podID="1e1a4c0b-7bda-4ada-8221-bd444286bd94" containerID="5e7f4e46b9ea4a2f89e35ca15fbbe81fcc34b6794fb39c937af2cebc4610bd97" exitCode=0 Oct 30 00:19:27.708274 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:27.708101 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f31177a-9fv92" event={"ID":"1e1a4c0b-7bda-4ada-8221-bd444286bd94","Type":"ContainerDied","Data":"5e7f4e46b9ea4a2f89e35ca15fbbe81fcc34b6794fb39c937af2cebc4610bd97"} Oct 30 00:19:28.749516 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:28.749476 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:19:28.750102 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:28.750081 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:19:28.751027 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:28.751004 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:19:28.901190 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:28.901160 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f31177a-9fv92" Oct 30 00:19:29.019840 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.019807 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e1a4c0b-7bda-4ada-8221-bd444286bd94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e1a4c0b-7bda-4ada-8221-bd444286bd94-maas-api-xks-ca\") pod \"1e1a4c0b-7bda-4ada-8221-bd444286bd94\" (UID: \"1e1a4c0b-7bda-4ada-8221-bd444286bd94\") " Oct 30 00:19:29.020017 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.019846 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e1a4c0b-7bda-4ada-8221-bd444286bd94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e1a4c0b-7bda-4ada-8221-bd444286bd94-maas-api-openshift-ca\") pod \"1e1a4c0b-7bda-4ada-8221-bd444286bd94\" (UID: \"1e1a4c0b-7bda-4ada-8221-bd444286bd94\") " Oct 30 00:19:29.020017 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.020013 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e1a4c0b-7bda-4ada-8221-bd444286bd94-kube-api-access-hzww9\" (UniqueName: \"kubernetes.io/projected/1e1a4c0b-7bda-4ada-8221-bd444286bd94-kube-api-access-hzww9\") pod \"1e1a4c0b-7bda-4ada-8221-bd444286bd94\" (UID: \"1e1a4c0b-7bda-4ada-8221-bd444286bd94\") " Oct 30 00:19:29.020306 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.020273 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e1a4c0b-7bda-4ada-8221-bd444286bd94-maas-api-openshift-ca" pod "1e1a4c0b-7bda-4ada-8221-bd444286bd94" (UID: "1e1a4c0b-7bda-4ada-8221-bd444286bd94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:19:29.020453 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.020424 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e1a4c0b-7bda-4ada-8221-bd444286bd94-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:19:29.022658 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.022628 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e1a4c0b-7bda-4ada-8221-bd444286bd94-maas-api-xks-ca" pod "1e1a4c0b-7bda-4ada-8221-bd444286bd94" (UID: "1e1a4c0b-7bda-4ada-8221-bd444286bd94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:19:29.022759 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.022632 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e1a4c0b-7bda-4ada-8221-bd444286bd94-kube-api-access-hzww9" pod "1e1a4c0b-7bda-4ada-8221-bd444286bd94" (UID: "1e1a4c0b-7bda-4ada-8221-bd444286bd94"). InnerVolumeSpecName "kube-api-access-hzww9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:19:29.121502 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.121473 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzww9\" (UniqueName: \"kubernetes.io/projected/1e1a4c0b-7bda-4ada-8221-bd444286bd94-kube-api-access-hzww9\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:19:29.121502 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.121498 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e1a4c0b-7bda-4ada-8221-bd444286bd94-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:19:29.728575 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.728543 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f31177a-9fv92" Oct 30 00:19:29.728575 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.728561 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f31177a-9fv92" event={"ID":"1e1a4c0b-7bda-4ada-8221-bd444286bd94","Type":"ContainerDied","Data":"4d600cf5e192b8eb70234c5598bfe1fe0e427d9ceda4b712e1e199d9b5996918"} Oct 30 00:19:29.728788 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.728610 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d600cf5e192b8eb70234c5598bfe1fe0e427d9ceda4b712e1e199d9b5996918" Oct 30 00:19:29.764063 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.764020 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34506eaf-rlg8l"] Oct 30 00:19:29.764762 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.764741 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1e1a4c0b-7bda-4ada-8221-bd444286bd94" containerName="revoke-keys" Oct 30 00:19:29.764762 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.764762 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e1a4c0b-7bda-4ada-8221-bd444286bd94" containerName="revoke-keys" Oct 30 00:19:29.764964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.764817 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d42bb9aa-fb91-462a-a01f-01c821cc2bb6" containerName="revoke-keys" Oct 30 00:19:29.764964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.764826 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d42bb9aa-fb91-462a-a01f-01c821cc2bb6" containerName="revoke-keys" Oct 30 00:19:29.765072 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.765009 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d42bb9aa-fb91-462a-a01f-01c821cc2bb6" containerName="revoke-keys" Oct 30 00:19:29.765072 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.765030 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e1a4c0b-7bda-4ada-8221-bd444286bd94" containerName="revoke-keys" Oct 30 00:19:29.768708 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.768688 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34506eaf-rlg8l" Oct 30 00:19:29.781982 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.781687 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:19:29.781982 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.781792 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:19:29.781982 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.781806 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:19:29.782221 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.781906 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:19:29.787205 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.787182 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34506eaf-rlg8l"] Oct 30 00:19:29.832569 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.832534 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b6h6\" (UniqueName: \"kubernetes.io/projected/273644c7-3e29-4731-81a9-fc1c0b7c4a4a-kube-api-access-5b6h6\") pod \"maas-api-revoke-sub-34506eaf-rlg8l\" (UID: \"273644c7-3e29-4731-81a9-fc1c0b7c4a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34506eaf-rlg8l" Oct 30 00:19:29.832740 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.832659 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/273644c7-3e29-4731-81a9-fc1c0b7c4a4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34506eaf-rlg8l\" (UID: \"273644c7-3e29-4731-81a9-fc1c0b7c4a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34506eaf-rlg8l" Oct 30 00:19:29.832740 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.832714 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/273644c7-3e29-4731-81a9-fc1c0b7c4a4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34506eaf-rlg8l\" (UID: \"273644c7-3e29-4731-81a9-fc1c0b7c4a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34506eaf-rlg8l" Oct 30 00:19:29.934123 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.934082 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/273644c7-3e29-4731-81a9-fc1c0b7c4a4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34506eaf-rlg8l\" (UID: \"273644c7-3e29-4731-81a9-fc1c0b7c4a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34506eaf-rlg8l" Oct 30 00:19:29.934295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.934264 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b6h6\" (UniqueName: \"kubernetes.io/projected/273644c7-3e29-4731-81a9-fc1c0b7c4a4a-kube-api-access-5b6h6\") pod \"maas-api-revoke-sub-34506eaf-rlg8l\" (UID: \"273644c7-3e29-4731-81a9-fc1c0b7c4a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34506eaf-rlg8l" Oct 30 00:19:29.934341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.934322 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/273644c7-3e29-4731-81a9-fc1c0b7c4a4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34506eaf-rlg8l\" (UID: \"273644c7-3e29-4731-81a9-fc1c0b7c4a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34506eaf-rlg8l" Oct 30 00:19:29.934784 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.934761 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/273644c7-3e29-4731-81a9-fc1c0b7c4a4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34506eaf-rlg8l\" (UID: \"273644c7-3e29-4731-81a9-fc1c0b7c4a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34506eaf-rlg8l" Oct 30 00:19:29.936633 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.936613 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/273644c7-3e29-4731-81a9-fc1c0b7c4a4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34506eaf-rlg8l\" (UID: \"273644c7-3e29-4731-81a9-fc1c0b7c4a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34506eaf-rlg8l" Oct 30 00:19:29.955400 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:29.955368 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b6h6\" (UniqueName: \"kubernetes.io/projected/273644c7-3e29-4731-81a9-fc1c0b7c4a4a-kube-api-access-5b6h6\") pod \"maas-api-revoke-sub-34506eaf-rlg8l\" (UID: \"273644c7-3e29-4731-81a9-fc1c0b7c4a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34506eaf-rlg8l" Oct 30 00:19:30.090533 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:30.090499 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34506eaf-rlg8l" Oct 30 00:19:30.393395 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:30.393365 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34506eaf-rlg8l"] Oct 30 00:19:30.394817 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:19:30.394795 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod273644c7_3e29_4731_81a9_fc1c0b7c4a4a.slice/crio-4080448dc0b1e42fbd4cb831feba9487326ca76fb9e81e775d3c4c58651ef2f1 WatchSource:0}: Error finding container 4080448dc0b1e42fbd4cb831feba9487326ca76fb9e81e775d3c4c58651ef2f1: Status 404 returned error can't find the container with id 4080448dc0b1e42fbd4cb831feba9487326ca76fb9e81e775d3c4c58651ef2f1 Oct 30 00:19:30.778933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:30.773981 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34506eaf-rlg8l" event={"ID":"273644c7-3e29-4731-81a9-fc1c0b7c4a4a","Type":"ContainerStarted","Data":"4080448dc0b1e42fbd4cb831feba9487326ca76fb9e81e775d3c4c58651ef2f1"} Oct 30 00:19:31.782160 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:31.782122 5001 generic.go:356] "Generic (PLEG): container finished" podID="273644c7-3e29-4731-81a9-fc1c0b7c4a4a" containerID="8c4e23d5a1341eda01baffa2b857efb5ce53b62e4c79e483ec75d4fe2e59a49b" exitCode=0 Oct 30 00:19:31.782577 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:31.782163 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34506eaf-rlg8l" event={"ID":"273644c7-3e29-4731-81a9-fc1c0b7c4a4a","Type":"ContainerDied","Data":"8c4e23d5a1341eda01baffa2b857efb5ce53b62e4c79e483ec75d4fe2e59a49b"} Oct 30 00:19:32.609977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:32.609944 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc"] Oct 30 00:19:32.615498 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:32.615467 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:19:32.634071 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:32.634045 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc"] Oct 30 00:19:32.637452 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:32.637430 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-7a7ac6\"/\"distinct-2-w1-7a7ac6-kserve-self-signed-certs\"" Oct 30 00:19:32.673146 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:32.673110 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-tmp-dir\") pod \"distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc\" (UID: \"2f0cfc05-3778-4a07-b822-cffc4d94ddb6\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:19:32.673487 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:32.673454 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndrvd\" (UniqueName: \"kubernetes.io/projected/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-kube-api-access-ndrvd\") pod \"distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc\" (UID: \"2f0cfc05-3778-4a07-b822-cffc4d94ddb6\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:19:32.673630 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:32.673517 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-tls-certs\") pod \"distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc\" (UID: \"2f0cfc05-3778-4a07-b822-cffc4d94ddb6\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:19:32.673693 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:32.673636 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-dshm\") pod \"distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc\" (UID: \"2f0cfc05-3778-4a07-b822-cffc4d94ddb6\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:19:32.673693 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:32.673678 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-home\") pod \"distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc\" (UID: \"2f0cfc05-3778-4a07-b822-cffc4d94ddb6\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:19:32.673806 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:32.673726 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-model-cache\") pod \"distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc\" (UID: \"2f0cfc05-3778-4a07-b822-cffc4d94ddb6\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:19:32.774993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:32.774954 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-tmp-dir\") pod \"distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc\" (UID: \"2f0cfc05-3778-4a07-b822-cffc4d94ddb6\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:19:32.775183 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:32.775088 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndrvd\" (UniqueName: \"kubernetes.io/projected/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-kube-api-access-ndrvd\") pod \"distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc\" (UID: \"2f0cfc05-3778-4a07-b822-cffc4d94ddb6\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:19:32.775183 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:32.775134 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-tls-certs\") pod \"distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc\" (UID: \"2f0cfc05-3778-4a07-b822-cffc4d94ddb6\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:19:32.775316 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:32.775200 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-dshm\") pod \"distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc\" (UID: \"2f0cfc05-3778-4a07-b822-cffc4d94ddb6\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:19:32.775316 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:32.775227 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-home\") pod \"distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc\" (UID: \"2f0cfc05-3778-4a07-b822-cffc4d94ddb6\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:19:32.775316 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:32.775268 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-model-cache\") pod \"distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc\" (UID: \"2f0cfc05-3778-4a07-b822-cffc4d94ddb6\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:19:32.775471 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:32.775372 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-tmp-dir\") pod \"distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc\" (UID: \"2f0cfc05-3778-4a07-b822-cffc4d94ddb6\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:19:32.775736 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:32.775707 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-model-cache\") pod \"distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc\" (UID: \"2f0cfc05-3778-4a07-b822-cffc4d94ddb6\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:19:32.775998 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:32.775973 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-home\") pod \"distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc\" (UID: \"2f0cfc05-3778-4a07-b822-cffc4d94ddb6\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:19:32.777560 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:32.777539 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-dshm\") pod \"distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc\" (UID: \"2f0cfc05-3778-4a07-b822-cffc4d94ddb6\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:19:32.778257 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:32.778238 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-tls-certs\") pod \"distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc\" (UID: \"2f0cfc05-3778-4a07-b822-cffc4d94ddb6\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:19:32.827944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:32.826748 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndrvd\" (UniqueName: \"kubernetes.io/projected/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-kube-api-access-ndrvd\") pod \"distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc\" (UID: \"2f0cfc05-3778-4a07-b822-cffc4d94ddb6\") " pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:19:32.934287 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:32.934188 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:19:33.240090 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:33.240065 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34506eaf-rlg8l" Oct 30 00:19:33.355471 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:33.355429 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/273644c7-3e29-4731-81a9-fc1c0b7c4a4a-kube-api-access-5b6h6\" (UniqueName: \"kubernetes.io/projected/273644c7-3e29-4731-81a9-fc1c0b7c4a4a-kube-api-access-5b6h6\") pod \"273644c7-3e29-4731-81a9-fc1c0b7c4a4a\" (UID: \"273644c7-3e29-4731-81a9-fc1c0b7c4a4a\") " Oct 30 00:19:33.355653 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:33.355480 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/273644c7-3e29-4731-81a9-fc1c0b7c4a4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/273644c7-3e29-4731-81a9-fc1c0b7c4a4a-maas-api-openshift-ca\") pod \"273644c7-3e29-4731-81a9-fc1c0b7c4a4a\" (UID: \"273644c7-3e29-4731-81a9-fc1c0b7c4a4a\") " Oct 30 00:19:33.355729 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:33.355649 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/273644c7-3e29-4731-81a9-fc1c0b7c4a4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/273644c7-3e29-4731-81a9-fc1c0b7c4a4a-maas-api-xks-ca\") pod \"273644c7-3e29-4731-81a9-fc1c0b7c4a4a\" (UID: \"273644c7-3e29-4731-81a9-fc1c0b7c4a4a\") " Oct 30 00:19:33.355890 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:33.355860 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/273644c7-3e29-4731-81a9-fc1c0b7c4a4a-maas-api-openshift-ca" pod "273644c7-3e29-4731-81a9-fc1c0b7c4a4a" (UID: "273644c7-3e29-4731-81a9-fc1c0b7c4a4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:19:33.356338 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:33.356314 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/273644c7-3e29-4731-81a9-fc1c0b7c4a4a-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:19:33.357677 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:33.357650 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/273644c7-3e29-4731-81a9-fc1c0b7c4a4a-kube-api-access-5b6h6" pod "273644c7-3e29-4731-81a9-fc1c0b7c4a4a" (UID: "273644c7-3e29-4731-81a9-fc1c0b7c4a4a"). InnerVolumeSpecName "kube-api-access-5b6h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:19:33.357753 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:33.357733 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/273644c7-3e29-4731-81a9-fc1c0b7c4a4a-maas-api-xks-ca" pod "273644c7-3e29-4731-81a9-fc1c0b7c4a4a" (UID: "273644c7-3e29-4731-81a9-fc1c0b7c4a4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:19:33.359118 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:33.359099 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc"] Oct 30 00:19:33.359850 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:19:33.359828 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f0cfc05_3778_4a07_b822_cffc4d94ddb6.slice/crio-ceee601082f5232937498db37807613a689f175d34b2c5b7e5ea19ca360db5c8 WatchSource:0}: Error finding container ceee601082f5232937498db37807613a689f175d34b2c5b7e5ea19ca360db5c8: Status 404 returned error can't find the container with id ceee601082f5232937498db37807613a689f175d34b2c5b7e5ea19ca360db5c8 Oct 30 00:19:33.457415 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:33.457383 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b6h6\" (UniqueName: \"kubernetes.io/projected/273644c7-3e29-4731-81a9-fc1c0b7c4a4a-kube-api-access-5b6h6\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:19:33.457415 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:33.457412 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/273644c7-3e29-4731-81a9-fc1c0b7c4a4a-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:19:33.932408 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:33.932113 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" event={"ID":"2f0cfc05-3778-4a07-b822-cffc4d94ddb6","Type":"ContainerStarted","Data":"14e4d2f1d6af626024fa624c224f87be29d12beb22b8cd096387bda75749abd0"} Oct 30 00:19:33.932408 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:33.932187 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" event={"ID":"2f0cfc05-3778-4a07-b822-cffc4d94ddb6","Type":"ContainerStarted","Data":"ceee601082f5232937498db37807613a689f175d34b2c5b7e5ea19ca360db5c8"} Oct 30 00:19:33.947698 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:33.937046 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34506eaf-rlg8l" event={"ID":"273644c7-3e29-4731-81a9-fc1c0b7c4a4a","Type":"ContainerDied","Data":"4080448dc0b1e42fbd4cb831feba9487326ca76fb9e81e775d3c4c58651ef2f1"} Oct 30 00:19:33.947698 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:33.937106 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4080448dc0b1e42fbd4cb831feba9487326ca76fb9e81e775d3c4c58651ef2f1" Oct 30 00:19:33.947698 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:33.937173 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34506eaf-rlg8l" Oct 30 00:19:33.947698 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:33.942348 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:19:33.974001 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:33.973947 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podStartSLOduration=1.973931391 podStartE2EDuration="1.973931391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:19:33.969294089 +0000 UTC m=+889.174795273" watchObservedRunningTime="2026-10-30 00:19:33.973931391 +0000 UTC m=+889.179432571" Oct 30 00:19:34.877106 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:34.877072 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n"] Oct 30 00:19:34.877669 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:34.877653 5001 cpu_manager.go:414] "containerMap: removing container" podUID="273644c7-3e29-4731-81a9-fc1c0b7c4a4a" containerName="revoke-keys" Oct 30 00:19:34.877669 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:34.877669 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="273644c7-3e29-4731-81a9-fc1c0b7c4a4a" containerName="revoke-keys" Oct 30 00:19:34.877803 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:34.877780 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="273644c7-3e29-4731-81a9-fc1c0b7c4a4a" containerName="revoke-keys" Oct 30 00:19:34.881666 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:34.881642 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:19:34.919499 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:34.919466 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n"] Oct 30 00:19:35.025591 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.025555 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf513d72-64d7-4444-ae0c-fda3aef69626-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:19:35.025977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.025616 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:19:35.025977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.025722 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf513d72-64d7-4444-ae0c-fda3aef69626-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:19:35.025977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.025768 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:19:35.025977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.025847 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:19:35.025977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.025887 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf513d72-64d7-4444-ae0c-fda3aef69626-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:19:35.025977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.025904 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrlt9\" (UniqueName: \"kubernetes.io/projected/cf513d72-64d7-4444-ae0c-fda3aef69626-kube-api-access-hrlt9\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:19:35.026180 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.026034 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:19:35.026180 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.026144 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:19:35.127524 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.127425 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:19:35.127524 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.127475 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf513d72-64d7-4444-ae0c-fda3aef69626-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:19:35.127524 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.127501 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrlt9\" (UniqueName: \"kubernetes.io/projected/cf513d72-64d7-4444-ae0c-fda3aef69626-kube-api-access-hrlt9\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:19:35.127814 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.127543 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:19:35.127814 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.127622 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:19:35.127814 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.127670 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf513d72-64d7-4444-ae0c-fda3aef69626-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:19:35.127814 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.127732 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:19:35.127814 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.127790 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf513d72-64d7-4444-ae0c-fda3aef69626-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:19:35.128121 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.127835 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:19:35.128121 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.127974 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:19:35.128978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.128274 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:19:35.128978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.128309 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:19:35.128978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.128368 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:19:35.128978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.128904 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf513d72-64d7-4444-ae0c-fda3aef69626-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:19:35.130481 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.130456 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:19:35.130622 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.130602 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf513d72-64d7-4444-ae0c-fda3aef69626-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:19:35.148635 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.148608 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrlt9\" (UniqueName: \"kubernetes.io/projected/cf513d72-64d7-4444-ae0c-fda3aef69626-kube-api-access-hrlt9\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:19:35.148782 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.148764 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf513d72-64d7-4444-ae0c-fda3aef69626-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:19:35.196690 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.196660 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:19:35.518878 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.518853 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n"] Oct 30 00:19:35.519959 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:19:35.519927 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf513d72_64d7_4444_ae0c_fda3aef69626.slice/crio-cea880e918e7cde8e89cd15f6a3ed4d2b6d7f899ad9107da2e124f677d4af536 WatchSource:0}: Error finding container cea880e918e7cde8e89cd15f6a3ed4d2b6d7f899ad9107da2e124f677d4af536: Status 404 returned error can't find the container with id cea880e918e7cde8e89cd15f6a3ed4d2b6d7f899ad9107da2e124f677d4af536 Oct 30 00:19:35.522347 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.522314 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:19:35.522441 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.522383 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:19:35.522441 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.522413 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:19:35.960868 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.960765 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" event={"ID":"cf513d72-64d7-4444-ae0c-fda3aef69626","Type":"ContainerStarted","Data":"730b7678f746047c06a2089e3202c50ab0d3ee50106b59e5e0364c215f155197"} Oct 30 00:19:35.960868 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:35.960825 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" event={"ID":"cf513d72-64d7-4444-ae0c-fda3aef69626","Type":"ContainerStarted","Data":"cea880e918e7cde8e89cd15f6a3ed4d2b6d7f899ad9107da2e124f677d4af536"} Oct 30 00:19:36.007356 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:36.007271 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" podStartSLOduration=2.007250773 podStartE2EDuration="2.007250773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:19:36.00146098 +0000 UTC m=+891.206962163" watchObservedRunningTime="2026-10-30 00:19:36.007250773 +0000 UTC m=+891.212751955" Oct 30 00:19:36.197213 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:36.197171 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:19:36.199138 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:36.199105 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 00:19:36.199301 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:36.199165 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" podUID="cf513d72-64d7-4444-ae0c-fda3aef69626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 00:19:37.197670 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:37.197637 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 00:19:37.198131 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:37.197690 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" podUID="cf513d72-64d7-4444-ae0c-fda3aef69626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 00:19:38.197271 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:38.197239 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 00:19:38.197425 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:38.197295 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" podUID="cf513d72-64d7-4444-ae0c-fda3aef69626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 00:19:39.198077 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:39.198038 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 00:19:39.198472 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:39.198096 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" podUID="cf513d72-64d7-4444-ae0c-fda3aef69626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 00:19:40.197451 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:40.197413 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 00:19:40.197641 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:40.197492 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" podUID="cf513d72-64d7-4444-ae0c-fda3aef69626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 00:19:41.197558 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:41.197520 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 00:19:41.198101 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:41.197590 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" podUID="cf513d72-64d7-4444-ae0c-fda3aef69626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 00:19:42.197965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:42.197899 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 00:19:42.198500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:42.197994 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" podUID="cf513d72-64d7-4444-ae0c-fda3aef69626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 00:19:42.269948 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:42.269898 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f08b449-d648r"] Oct 30 00:19:42.275778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:42.275753 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f08b449-d648r" Oct 30 00:19:42.294720 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:42.294689 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f08b449-d648r"] Oct 30 00:19:42.297341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:42.297319 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:19:42.297574 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:42.297559 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:19:42.297804 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:42.297790 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:19:42.298014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:42.298000 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:19:42.369254 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:42.369221 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/824446e0-437d-4fd3-902c-5afdfacda635-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f08b449-d648r\" (UID: \"824446e0-437d-4fd3-902c-5afdfacda635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f08b449-d648r" Oct 30 00:19:42.369412 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:42.369280 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkjr7\" (UniqueName: \"kubernetes.io/projected/824446e0-437d-4fd3-902c-5afdfacda635-kube-api-access-qkjr7\") pod \"maas-api-revoke-sub-9f08b449-d648r\" (UID: \"824446e0-437d-4fd3-902c-5afdfacda635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f08b449-d648r" Oct 30 00:19:42.369495 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:42.369467 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/824446e0-437d-4fd3-902c-5afdfacda635-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f08b449-d648r\" (UID: \"824446e0-437d-4fd3-902c-5afdfacda635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f08b449-d648r" Oct 30 00:19:42.470719 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:42.470610 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/824446e0-437d-4fd3-902c-5afdfacda635-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f08b449-d648r\" (UID: \"824446e0-437d-4fd3-902c-5afdfacda635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f08b449-d648r" Oct 30 00:19:42.470719 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:42.470693 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkjr7\" (UniqueName: \"kubernetes.io/projected/824446e0-437d-4fd3-902c-5afdfacda635-kube-api-access-qkjr7\") pod \"maas-api-revoke-sub-9f08b449-d648r\" (UID: \"824446e0-437d-4fd3-902c-5afdfacda635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f08b449-d648r" Oct 30 00:19:42.471021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:42.470760 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/824446e0-437d-4fd3-902c-5afdfacda635-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f08b449-d648r\" (UID: \"824446e0-437d-4fd3-902c-5afdfacda635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f08b449-d648r" Oct 30 00:19:42.471773 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:42.471746 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/824446e0-437d-4fd3-902c-5afdfacda635-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f08b449-d648r\" (UID: \"824446e0-437d-4fd3-902c-5afdfacda635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f08b449-d648r" Oct 30 00:19:42.473725 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:42.473693 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/824446e0-437d-4fd3-902c-5afdfacda635-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f08b449-d648r\" (UID: \"824446e0-437d-4fd3-902c-5afdfacda635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f08b449-d648r" Oct 30 00:19:42.505274 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:42.505250 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkjr7\" (UniqueName: \"kubernetes.io/projected/824446e0-437d-4fd3-902c-5afdfacda635-kube-api-access-qkjr7\") pod \"maas-api-revoke-sub-9f08b449-d648r\" (UID: \"824446e0-437d-4fd3-902c-5afdfacda635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f08b449-d648r" Oct 30 00:19:42.591736 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:42.591697 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f08b449-d648r" Oct 30 00:19:42.906923 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:42.906881 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f08b449-d648r"] Oct 30 00:19:42.908344 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:19:42.908321 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod824446e0_437d_4fd3_902c_5afdfacda635.slice/crio-d4dbdda9e21251fc20dafb3ff0ebb56aa326f28faa3be7d433dc792df6ef069d WatchSource:0}: Error finding container d4dbdda9e21251fc20dafb3ff0ebb56aa326f28faa3be7d433dc792df6ef069d: Status 404 returned error can't find the container with id d4dbdda9e21251fc20dafb3ff0ebb56aa326f28faa3be7d433dc792df6ef069d Oct 30 00:19:42.934963 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:42.934940 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:19:42.946984 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:42.946961 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:19:43.059474 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:43.059435 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f08b449-d648r" event={"ID":"824446e0-437d-4fd3-902c-5afdfacda635","Type":"ContainerStarted","Data":"d4dbdda9e21251fc20dafb3ff0ebb56aa326f28faa3be7d433dc792df6ef069d"} Oct 30 00:19:43.070794 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:43.070770 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:19:43.202362 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:43.202288 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:19:43.202727 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:43.202669 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:19:43.203323 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:43.203306 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:19:44.069905 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.069816 5001 generic.go:356] "Generic (PLEG): container finished" podID="824446e0-437d-4fd3-902c-5afdfacda635" containerID="3c0688d39574f44f040ef9e0d45573a3de277ca220435ba8d8f80927cebaf6ba" exitCode=0 Oct 30 00:19:44.069905 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.069883 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f08b449-d648r" event={"ID":"824446e0-437d-4fd3-902c-5afdfacda635","Type":"ContainerDied","Data":"3c0688d39574f44f040ef9e0d45573a3de277ca220435ba8d8f80927cebaf6ba"} Oct 30 00:19:44.249635 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.249596 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd"] Oct 30 00:19:44.254046 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.254012 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:19:44.286672 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.286644 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd"] Oct 30 00:19:44.401027 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.400939 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:19:44.401027 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.401006 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:19:44.401218 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.401073 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:19:44.401218 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.401129 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:19:44.401218 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.401193 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnpzj\" (UniqueName: \"kubernetes.io/projected/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-kube-api-access-dnpzj\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:19:44.401341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.401323 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:19:44.401432 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.401364 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:19:44.401432 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.401390 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:19:44.401533 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.401508 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:19:44.502492 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.502456 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:19:44.502679 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.502503 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:19:44.502679 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.502528 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:19:44.502679 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.502579 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:19:44.502679 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.502616 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:19:44.502679 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.502642 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:19:44.502679 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.502657 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:19:44.502679 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.502678 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:19:44.503106 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.502755 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnpzj\" (UniqueName: \"kubernetes.io/projected/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-kube-api-access-dnpzj\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:19:44.503106 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.502943 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:19:44.503106 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.502964 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:19:44.503106 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.503074 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:19:44.503391 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.503372 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:19:44.503468 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.503447 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:19:44.505042 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.505019 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:19:44.505452 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.505423 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:19:44.523154 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.523131 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:19:44.523516 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.523488 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnpzj\" (UniqueName: \"kubernetes.io/projected/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-kube-api-access-dnpzj\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:19:44.575324 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.575294 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:19:44.894348 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.894326 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd"] Oct 30 00:19:44.895983 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:19:44.895945 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5ff3159_b9e7_4865_b35a_c2051d7e5c20.slice/crio-0a6d036aec85bc7d5adf83f6dc4863fbbd50328cb475235c65c6c3b57d43eeb0 WatchSource:0}: Error finding container 0a6d036aec85bc7d5adf83f6dc4863fbbd50328cb475235c65c6c3b57d43eeb0: Status 404 returned error can't find the container with id 0a6d036aec85bc7d5adf83f6dc4863fbbd50328cb475235c65c6c3b57d43eeb0 Oct 30 00:19:44.898275 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.898242 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:19:44.898388 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.898302 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:19:44.898388 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:44.898333 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:19:45.081692 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:45.081654 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" event={"ID":"a5ff3159-b9e7-4865-b35a-c2051d7e5c20","Type":"ContainerStarted","Data":"858956fd8fd665f36b2412e601621014cccbdbd473f21c4f2fdf5ce73b7a894c"} Oct 30 00:19:45.081881 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:45.081709 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" event={"ID":"a5ff3159-b9e7-4865-b35a-c2051d7e5c20","Type":"ContainerStarted","Data":"0a6d036aec85bc7d5adf83f6dc4863fbbd50328cb475235c65c6c3b57d43eeb0"} Oct 30 00:19:45.125798 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:45.123266 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" podStartSLOduration=1.123245909 podStartE2EDuration="1.123245909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:19:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:19:45.122386248 +0000 UTC m=+900.327887430" watchObservedRunningTime="2026-10-30 00:19:45.123245909 +0000 UTC m=+900.328747090" Oct 30 00:19:45.156781 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:45.156753 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8gb5_9fa688d8-29ca-4c1f-926a-f1a4099e6b44/console-operator/2.log" Oct 30 00:19:45.157253 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:45.157232 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8gb5_9fa688d8-29ca-4c1f-926a-f1a4099e6b44/console-operator/2.log" Oct 30 00:19:45.247751 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:45.247724 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f08b449-d648r" Oct 30 00:19:45.313983 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:45.313906 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/824446e0-437d-4fd3-902c-5afdfacda635-kube-api-access-qkjr7\" (UniqueName: \"kubernetes.io/projected/824446e0-437d-4fd3-902c-5afdfacda635-kube-api-access-qkjr7\") pod \"824446e0-437d-4fd3-902c-5afdfacda635\" (UID: \"824446e0-437d-4fd3-902c-5afdfacda635\") " Oct 30 00:19:45.314465 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:45.314010 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/824446e0-437d-4fd3-902c-5afdfacda635-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/824446e0-437d-4fd3-902c-5afdfacda635-maas-api-xks-ca\") pod \"824446e0-437d-4fd3-902c-5afdfacda635\" (UID: \"824446e0-437d-4fd3-902c-5afdfacda635\") " Oct 30 00:19:45.314465 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:45.314131 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/824446e0-437d-4fd3-902c-5afdfacda635-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/824446e0-437d-4fd3-902c-5afdfacda635-maas-api-openshift-ca\") pod \"824446e0-437d-4fd3-902c-5afdfacda635\" (UID: \"824446e0-437d-4fd3-902c-5afdfacda635\") " Oct 30 00:19:45.314661 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:45.314631 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/824446e0-437d-4fd3-902c-5afdfacda635-maas-api-openshift-ca" pod "824446e0-437d-4fd3-902c-5afdfacda635" (UID: "824446e0-437d-4fd3-902c-5afdfacda635"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:19:45.314814 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:45.314794 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/824446e0-437d-4fd3-902c-5afdfacda635-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:19:45.316682 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:45.316655 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/824446e0-437d-4fd3-902c-5afdfacda635-maas-api-xks-ca" pod "824446e0-437d-4fd3-902c-5afdfacda635" (UID: "824446e0-437d-4fd3-902c-5afdfacda635"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:19:45.316993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:45.316962 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824446e0-437d-4fd3-902c-5afdfacda635-kube-api-access-qkjr7" pod "824446e0-437d-4fd3-902c-5afdfacda635" (UID: "824446e0-437d-4fd3-902c-5afdfacda635"). InnerVolumeSpecName "kube-api-access-qkjr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:19:45.416229 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:45.416156 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkjr7\" (UniqueName: \"kubernetes.io/projected/824446e0-437d-4fd3-902c-5afdfacda635-kube-api-access-qkjr7\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:19:45.416229 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:45.416183 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/824446e0-437d-4fd3-902c-5afdfacda635-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:19:45.575642 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:45.575611 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:19:45.577105 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:45.577083 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:19:45.577204 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:45.577131 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" podUID="a5ff3159-b9e7-4865-b35a-c2051d7e5c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 00:19:46.094212 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:46.094183 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f08b449-d648r" Oct 30 00:19:46.094212 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:46.094202 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f08b449-d648r" event={"ID":"824446e0-437d-4fd3-902c-5afdfacda635","Type":"ContainerDied","Data":"d4dbdda9e21251fc20dafb3ff0ebb56aa326f28faa3be7d433dc792df6ef069d"} Oct 30 00:19:46.094468 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:46.094256 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4dbdda9e21251fc20dafb3ff0ebb56aa326f28faa3be7d433dc792df6ef069d" Oct 30 00:19:46.575663 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:46.575628 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:19:46.576095 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:46.575686 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" podUID="a5ff3159-b9e7-4865-b35a-c2051d7e5c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 00:19:47.398188 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:19:47.398114 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:19:47.578582 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:47.578536 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:19:47.579235 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:47.579181 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" podUID="a5ff3159-b9e7-4865-b35a-c2051d7e5c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 00:19:48.580936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:48.578293 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:19:48.580936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:48.578372 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" podUID="a5ff3159-b9e7-4865-b35a-c2051d7e5c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 00:19:49.342934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.324693 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk"] Oct 30 00:19:49.355220 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.351738 5001 cpu_manager.go:414] "containerMap: removing container" podUID="824446e0-437d-4fd3-902c-5afdfacda635" containerName="revoke-keys" Oct 30 00:19:49.355220 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.351769 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="824446e0-437d-4fd3-902c-5afdfacda635" containerName="revoke-keys" Oct 30 00:19:49.355220 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.351986 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="824446e0-437d-4fd3-902c-5afdfacda635" containerName="revoke-keys" Oct 30 00:19:49.391932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.389258 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:19:49.481323 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.473986 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk"] Oct 30 00:19:49.560713 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.560670 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-credential-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:19:49.561024 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.561005 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e723059f-3ca0-45b1-a035-6ce96939fe4b-istiod-ca-cert\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:19:49.601992 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.591042 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e723059f-3ca0-45b1-a035-6ce96939fe4b-istio-token\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:19:49.601992 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.591140 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-istio-envoy\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:19:49.601992 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.591222 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-istio-data\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:19:49.601992 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.591336 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-workload-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:19:49.601992 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.591387 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e723059f-3ca0-45b1-a035-6ce96939fe4b-istio-podinfo\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:19:49.601992 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.591442 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-workload-certs\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:19:49.601992 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.591541 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wpqm\" (UniqueName: \"kubernetes.io/projected/e723059f-3ca0-45b1-a035-6ce96939fe4b-kube-api-access-4wpqm\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:19:49.601992 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.576393 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:19:49.601992 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.592157 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" podUID="a5ff3159-b9e7-4865-b35a-c2051d7e5c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 00:19:49.738940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.706602 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-workload-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:19:49.739292 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.739260 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e723059f-3ca0-45b1-a035-6ce96939fe4b-istio-podinfo\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:19:49.743931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.739434 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-workload-certs\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:19:49.743931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.739530 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wpqm\" (UniqueName: \"kubernetes.io/projected/e723059f-3ca0-45b1-a035-6ce96939fe4b-kube-api-access-4wpqm\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:19:49.743931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.739595 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-credential-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:19:49.743931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.739683 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e723059f-3ca0-45b1-a035-6ce96939fe4b-istiod-ca-cert\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:19:49.743931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.739791 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e723059f-3ca0-45b1-a035-6ce96939fe4b-istio-token\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:19:49.743931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.739852 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-istio-envoy\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:19:49.753952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.719998 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-workload-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:19:49.754224 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.754193 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-istio-data\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:19:49.770967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.754742 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-istio-data\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:19:49.806051 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.806016 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-credential-socket\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:19:49.806546 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.806522 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-workload-certs\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:19:49.809275 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.807538 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e723059f-3ca0-45b1-a035-6ce96939fe4b-istiod-ca-cert\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:19:49.851896 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.851851 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wpqm\" (UniqueName: \"kubernetes.io/projected/e723059f-3ca0-45b1-a035-6ce96939fe4b-kube-api-access-4wpqm\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:19:49.855980 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.852079 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e723059f-3ca0-45b1-a035-6ce96939fe4b-istio-token\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:19:49.886468 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.886395 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-istio-envoy\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:19:49.888029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.887990 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e723059f-3ca0-45b1-a035-6ce96939fe4b-istio-podinfo\") pod \"e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:19:49.955507 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:49.955188 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:19:50.063554 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:50.063016 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:50.085637 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:50.084998 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:50.220992 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:50.209000 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:50.253399 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:50.253016 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:50.521513 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:50.508058 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 00:19:50.590424 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:50.590192 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:19:50.590424 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:50.590253 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" podUID="a5ff3159-b9e7-4865-b35a-c2051d7e5c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 00:19:51.080957 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:51.064059 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 30 00:19:51.111532 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:51.083407 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:19:51.199775 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:51.199719 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:19:51.200098 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:51.200070 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:19:51.200289 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:51.200261 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:19:51.318064 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:51.318023 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk"] Oct 30 00:19:51.577055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:51.577021 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:19:51.581065 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:51.581021 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" podUID="a5ff3159-b9e7-4865-b35a-c2051d7e5c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 00:19:51.604953 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:51.593658 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" event={"ID":"e723059f-3ca0-45b1-a035-6ce96939fe4b","Type":"ContainerStarted","Data":"83e99d235bf8248957c960b446fb68bb9fbc6d6f28d2b740fc3b9982c258f70a"} Oct 30 00:19:51.813956 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:51.809006 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:19:51.850992 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:51.847009 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:52.093938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:52.081147 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 30 00:19:52.093938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:52.092232 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:52.118988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:52.097249 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:52.217938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:52.204002 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:52.331429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:52.331393 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:52.515463 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:52.515413 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:52.627813 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:52.609072 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:19:52.627813 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:52.609137 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" podUID="a5ff3159-b9e7-4865-b35a-c2051d7e5c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 00:19:52.831323 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:52.824329 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:19:52.902748 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:52.892869 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:19:52.902748 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:52.893488 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:52.902748 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:52.902381 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:19:52.903068 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:52.902927 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:19:53.129320 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:53.129108 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:53.308040 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:53.307995 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 00:19:53.392163 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:53.131718 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:53.453937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:53.438250 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:53.605937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:53.580519 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:53.822979 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:53.810997 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:19:53.823276 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:53.823237 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" podUID="a5ff3159-b9e7-4865-b35a-c2051d7e5c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 00:19:53.846009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:53.834260 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:19:53.950937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:53.895557 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:53.950937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:53.895591 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:53.950937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:53.908631 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" event={"ID":"e723059f-3ca0-45b1-a035-6ce96939fe4b","Type":"ContainerStarted","Data":"3afb7dc28f6285c6b85072f7c8d2474f1a30607fc0a6fca1ec087c9231579375"} Oct 30 00:19:53.950937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:53.943428 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:19:53.950937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:53.943476 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:19:53.950937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:53.943606 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:19:53.950937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:53.943641 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:19:53.950937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:53.943676 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:19:53.956924 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:53.956871 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:19:53.989347 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:53.964672 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 00:19:53.997025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:53.996983 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" podUID="e723059f-3ca0-45b1-a035-6ce96939fe4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 00:19:54.127129 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:54.127034 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:54.267937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:54.205695 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:54.316561 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:54.310990 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 00:19:54.367949 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:54.345694 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:19:54.367949 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:54.345752 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" podUID="35bc1599-694f-4f89-9a8a-7667310b9b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:54.432031 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:54.413638 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:54.463988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:54.461712 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:54.595590 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:54.590581 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:19:54.595590 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:54.590646 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" podUID="a5ff3159-b9e7-4865-b35a-c2051d7e5c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 00:19:54.805123 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:54.805081 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 00:19:54.977331 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:54.977286 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:54.977989 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:54.977956 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:19:54.978429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:54.978398 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:19:55.170818 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.170721 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 00:19:55.171123 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.171075 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" podUID="e723059f-3ca0-45b1-a035-6ce96939fe4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 00:19:55.350562 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.330066 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:55.353531 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.353494 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:55.417073 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.417022 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:55.426041 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.425118 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" podStartSLOduration=6.425101426 podStartE2EDuration="6.425101426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:19:53.906300375 +0000 UTC m=+909.111801553" watchObservedRunningTime="2026-10-30 00:19:55.425101426 +0000 UTC m=+910.630602608" Oct 30 00:19:55.460153 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.460122 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht"] Oct 30 00:19:55.547117 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.547081 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:19:55.547301 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.547128 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht"] Oct 30 00:19:55.547301 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.547248 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:19:55.548246 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.548142 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:19:55.592596 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.592555 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:19:55.594168 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.594100 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:19:55.594168 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.594156 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" podUID="a5ff3159-b9e7-4865-b35a-c2051d7e5c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 00:19:55.627936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.625104 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:19:55.627936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.626030 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:19:55.629846 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.629817 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istio-podinfo\") pod \"e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:19:55.634169 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.634141 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66wk5\" (UniqueName: \"kubernetes.io/projected/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-kube-api-access-66wk5\") pod \"e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:19:55.634352 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.634335 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-workload-certs\") pod \"e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:19:55.634645 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.634627 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-workload-socket\") pod \"e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:19:55.634847 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.634828 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istio-data\") pod \"e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:19:55.634990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.634976 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istiod-ca-cert\") pod \"e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:19:55.635111 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.635095 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-credential-socket\") pod \"e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:19:55.635390 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.635375 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istio-token\") pod \"e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:19:55.635502 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.635489 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istio-envoy\") pod \"e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:19:55.709468 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.689102 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-3fae6d-openshift-default-dockercfg-fvxbl\"" Oct 30 00:19:55.758973 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.758931 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66wk5\" (UniqueName: \"kubernetes.io/projected/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-kube-api-access-66wk5\") pod \"e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:19:55.764935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.759226 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-workload-certs\") pod \"e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:19:55.765417 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.765298 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-workload-socket\") pod \"e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:19:55.779058 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.779015 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istio-data\") pod \"e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:19:55.779279 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.779257 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istiod-ca-cert\") pod \"e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:19:55.779427 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.779410 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-credential-socket\") pod \"e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:19:55.779644 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.779627 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istio-token\") pod \"e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:19:55.779766 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.779747 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istio-envoy\") pod \"e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:19:55.780012 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.779993 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istio-podinfo\") pod \"e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:19:55.815157 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.784779 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istio-data\") pod \"e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:19:55.836066 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.799755 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-credential-socket\") pod \"e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:19:55.836066 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.800476 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istiod-ca-cert\") pod \"e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:19:55.836066 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.801550 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-workload-certs\") pod \"e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:19:55.836066 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.802135 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-workload-socket\") pod \"e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:19:55.836066 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.822772 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:55.877033 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.877003 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istio-podinfo\") pod \"e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:19:55.878800 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.878773 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istio-envoy\") pod \"e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:19:55.957556 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.957526 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 00:19:55.957811 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.957783 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" podUID="e723059f-3ca0-45b1-a035-6ce96939fe4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 00:19:55.969925 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.969839 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istio-token\") pod \"e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:19:55.979196 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.979164 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:19:55.979645 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:55.979619 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:56.256994 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:56.235482 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66wk5\" (UniqueName: \"kubernetes.io/projected/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-kube-api-access-66wk5\") pod \"e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:19:56.292881 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:56.292847 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:19:56.628580 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:56.628490 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:56.659532 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:56.655406 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:19:56.674769 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:56.669394 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:19:56.674769 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:56.669454 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" podUID="a5ff3159-b9e7-4865-b35a-c2051d7e5c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 00:19:56.707094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:56.707050 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:56.744758 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:56.709028 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:19:56.956966 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:56.956856 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 00:19:56.957272 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:56.957239 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" podUID="e723059f-3ca0-45b1-a035-6ce96939fe4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 00:19:57.018352 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:56.992467 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:57.199937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:57.190034 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:19:57.793067 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:19:57.793034 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf42670c1_7716_46d8_9bc3_3f06cf8caa8a.slice/crio-65d96276fe8706cd9dc26b290762928fbe422a2c024b1206faefa31549019cb4 WatchSource:0}: Error finding container 65d96276fe8706cd9dc26b290762928fbe422a2c024b1206faefa31549019cb4: Status 404 returned error can't find the container with id 65d96276fe8706cd9dc26b290762928fbe422a2c024b1206faefa31549019cb4 Oct 30 00:19:57.793795 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:57.793763 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht"] Oct 30 00:19:57.797020 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:57.796723 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:19:57.797020 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:57.796829 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:19:57.797020 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:57.796894 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:19:57.960967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:57.960602 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 00:19:57.960967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:57.960660 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" podUID="e723059f-3ca0-45b1-a035-6ce96939fe4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 00:19:58.000109 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:57.999859 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" event={"ID":"f42670c1-7716-46d8-9bc3-3f06cf8caa8a","Type":"ContainerStarted","Data":"65d96276fe8706cd9dc26b290762928fbe422a2c024b1206faefa31549019cb4"} Oct 30 00:19:58.577001 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:58.576963 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:19:58.577199 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:58.577027 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" podUID="a5ff3159-b9e7-4865-b35a-c2051d7e5c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:58.957757 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:58.957263 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 00:19:58.957757 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:58.957325 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" podUID="e723059f-3ca0-45b1-a035-6ce96939fe4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 00:19:59.068935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:59.059496 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" event={"ID":"f42670c1-7716-46d8-9bc3-3f06cf8caa8a","Type":"ContainerStarted","Data":"5228870d920b4c45c868cb243140a6e055def8b47c6d8277afa434d652ba9250"} Oct 30 00:19:59.166937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:59.162297 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" podStartSLOduration=6.162272106 podStartE2EDuration="6.162272106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:19:59.117514837 +0000 UTC m=+914.323016018" watchObservedRunningTime="2026-10-30 00:19:59.162272106 +0000 UTC m=+914.367773288" Oct 30 00:19:59.308954 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:59.306993 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:19:59.333934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:59.319261 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:19:59.576020 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:59.575892 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:19:59.618209 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:59.618173 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:19:59.618484 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:59.618450 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" podUID="a5ff3159-b9e7-4865-b35a-c2051d7e5c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:19:59.930986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:59.922456 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe23b4ca-pcwj9"] Oct 30 00:19:59.974600 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:59.956274 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 00:19:59.974600 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:59.956354 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" podUID="e723059f-3ca0-45b1-a035-6ce96939fe4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 00:19:59.980540 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:59.980497 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe23b4ca-pcwj9" Oct 30 00:19:59.983776 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:59.983748 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe23b4ca-pcwj9"] Oct 30 00:19:59.994347 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:19:59.994324 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:20:00.000046 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:00.000025 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:20:00.004206 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:00.004188 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:20:00.004264 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:00.004188 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:20:00.053678 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:00.052311 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e917f1ed-a050-40ef-b78e-43f1fbdbf461-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe23b4ca-pcwj9\" (UID: \"e917f1ed-a050-40ef-b78e-43f1fbdbf461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe23b4ca-pcwj9" Oct 30 00:20:00.053678 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:00.052519 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e917f1ed-a050-40ef-b78e-43f1fbdbf461-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe23b4ca-pcwj9\" (UID: \"e917f1ed-a050-40ef-b78e-43f1fbdbf461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe23b4ca-pcwj9" Oct 30 00:20:00.053678 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:00.052786 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfckh\" (UniqueName: \"kubernetes.io/projected/e917f1ed-a050-40ef-b78e-43f1fbdbf461-kube-api-access-wfckh\") pod \"maas-api-revoke-sub-fe23b4ca-pcwj9\" (UID: \"e917f1ed-a050-40ef-b78e-43f1fbdbf461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe23b4ca-pcwj9" Oct 30 00:20:00.071703 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:00.071637 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:20:00.072808 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:00.072778 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:20:00.163662 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:00.163078 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e917f1ed-a050-40ef-b78e-43f1fbdbf461-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe23b4ca-pcwj9\" (UID: \"e917f1ed-a050-40ef-b78e-43f1fbdbf461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe23b4ca-pcwj9" Oct 30 00:20:00.164946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:00.164502 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfckh\" (UniqueName: \"kubernetes.io/projected/e917f1ed-a050-40ef-b78e-43f1fbdbf461-kube-api-access-wfckh\") pod \"maas-api-revoke-sub-fe23b4ca-pcwj9\" (UID: \"e917f1ed-a050-40ef-b78e-43f1fbdbf461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe23b4ca-pcwj9" Oct 30 00:20:00.164946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:00.164618 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e917f1ed-a050-40ef-b78e-43f1fbdbf461-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe23b4ca-pcwj9\" (UID: \"e917f1ed-a050-40ef-b78e-43f1fbdbf461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe23b4ca-pcwj9" Oct 30 00:20:00.168050 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:00.168000 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e917f1ed-a050-40ef-b78e-43f1fbdbf461-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe23b4ca-pcwj9\" (UID: \"e917f1ed-a050-40ef-b78e-43f1fbdbf461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe23b4ca-pcwj9" Oct 30 00:20:00.176381 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:00.176333 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e917f1ed-a050-40ef-b78e-43f1fbdbf461-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe23b4ca-pcwj9\" (UID: \"e917f1ed-a050-40ef-b78e-43f1fbdbf461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe23b4ca-pcwj9" Oct 30 00:20:00.187424 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:00.186811 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfckh\" (UniqueName: \"kubernetes.io/projected/e917f1ed-a050-40ef-b78e-43f1fbdbf461-kube-api-access-wfckh\") pod \"maas-api-revoke-sub-fe23b4ca-pcwj9\" (UID: \"e917f1ed-a050-40ef-b78e-43f1fbdbf461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe23b4ca-pcwj9" Oct 30 00:20:00.301709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:00.301672 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe23b4ca-pcwj9" Oct 30 00:20:00.348940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:00.348178 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:20:00.356390 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:00.356321 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:20:00.691555 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:00.691516 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe23b4ca-pcwj9"] Oct 30 00:20:00.695443 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:20:00.695416 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode917f1ed_a050_40ef_b78e_43f1fbdbf461.slice/crio-9c784844e377c1f1058d901ef9a973bdd5d6b7fcad98f0258a59bb92489e2778 WatchSource:0}: Error finding container 9c784844e377c1f1058d901ef9a973bdd5d6b7fcad98f0258a59bb92489e2778: Status 404 returned error can't find the container with id 9c784844e377c1f1058d901ef9a973bdd5d6b7fcad98f0258a59bb92489e2778 Oct 30 00:20:00.956040 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:00.955973 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 00:20:00.956040 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:00.956027 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" podUID="e723059f-3ca0-45b1-a035-6ce96939fe4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 00:20:01.080732 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:01.080699 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe23b4ca-pcwj9" event={"ID":"e917f1ed-a050-40ef-b78e-43f1fbdbf461","Type":"ContainerStarted","Data":"9c784844e377c1f1058d901ef9a973bdd5d6b7fcad98f0258a59bb92489e2778"} Oct 30 00:20:01.956053 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:01.956012 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 00:20:01.956224 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:01.956071 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" podUID="e723059f-3ca0-45b1-a035-6ce96939fe4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 00:20:02.090579 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:02.090483 5001 generic.go:356] "Generic (PLEG): container finished" podID="e917f1ed-a050-40ef-b78e-43f1fbdbf461" containerID="6804f1981876a6b16fb1d607ef3b306f901c947c9f5f2dd6428d0adac5b71ab0" exitCode=0 Oct 30 00:20:02.090976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:02.090578 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe23b4ca-pcwj9" event={"ID":"e917f1ed-a050-40ef-b78e-43f1fbdbf461","Type":"ContainerDied","Data":"6804f1981876a6b16fb1d607ef3b306f901c947c9f5f2dd6428d0adac5b71ab0"} Oct 30 00:20:03.325236 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:03.325206 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:20:03.325723 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:03.325613 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:20:03.326860 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:03.326838 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:20:03.332446 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:03.331504 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe23b4ca-pcwj9" Oct 30 00:20:03.467986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:03.447323 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e917f1ed-a050-40ef-b78e-43f1fbdbf461-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e917f1ed-a050-40ef-b78e-43f1fbdbf461-maas-api-openshift-ca\") pod \"e917f1ed-a050-40ef-b78e-43f1fbdbf461\" (UID: \"e917f1ed-a050-40ef-b78e-43f1fbdbf461\") " Oct 30 00:20:03.467986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:03.447449 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e917f1ed-a050-40ef-b78e-43f1fbdbf461-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e917f1ed-a050-40ef-b78e-43f1fbdbf461-maas-api-xks-ca\") pod \"e917f1ed-a050-40ef-b78e-43f1fbdbf461\" (UID: \"e917f1ed-a050-40ef-b78e-43f1fbdbf461\") " Oct 30 00:20:03.467986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:03.447615 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e917f1ed-a050-40ef-b78e-43f1fbdbf461-kube-api-access-wfckh\" (UniqueName: \"kubernetes.io/projected/e917f1ed-a050-40ef-b78e-43f1fbdbf461-kube-api-access-wfckh\") pod \"e917f1ed-a050-40ef-b78e-43f1fbdbf461\" (UID: \"e917f1ed-a050-40ef-b78e-43f1fbdbf461\") " Oct 30 00:20:03.542787 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:03.518968 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e917f1ed-a050-40ef-b78e-43f1fbdbf461-maas-api-openshift-ca" pod "e917f1ed-a050-40ef-b78e-43f1fbdbf461" (UID: "e917f1ed-a050-40ef-b78e-43f1fbdbf461"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:20:03.542787 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:03.542087 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e917f1ed-a050-40ef-b78e-43f1fbdbf461-kube-api-access-wfckh" pod "e917f1ed-a050-40ef-b78e-43f1fbdbf461" (UID: "e917f1ed-a050-40ef-b78e-43f1fbdbf461"). InnerVolumeSpecName "kube-api-access-wfckh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:20:03.561933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:03.546591 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e917f1ed-a050-40ef-b78e-43f1fbdbf461-maas-api-xks-ca" pod "e917f1ed-a050-40ef-b78e-43f1fbdbf461" (UID: "e917f1ed-a050-40ef-b78e-43f1fbdbf461"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:20:03.593683 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:03.573034 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfckh\" (UniqueName: \"kubernetes.io/projected/e917f1ed-a050-40ef-b78e-43f1fbdbf461-kube-api-access-wfckh\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:20:03.593683 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:03.573074 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e917f1ed-a050-40ef-b78e-43f1fbdbf461-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:20:03.593683 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:03.573091 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e917f1ed-a050-40ef-b78e-43f1fbdbf461-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:20:04.144107 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:04.144078 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe23b4ca-pcwj9" Oct 30 00:20:04.144355 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:04.144140 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe23b4ca-pcwj9" event={"ID":"e917f1ed-a050-40ef-b78e-43f1fbdbf461","Type":"ContainerDied","Data":"9c784844e377c1f1058d901ef9a973bdd5d6b7fcad98f0258a59bb92489e2778"} Oct 30 00:20:04.144502 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:04.144372 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c784844e377c1f1058d901ef9a973bdd5d6b7fcad98f0258a59bb92489e2778" Oct 30 00:20:05.978067 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:05.978038 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6"] Oct 30 00:20:05.978623 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:05.978609 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e917f1ed-a050-40ef-b78e-43f1fbdbf461" containerName="revoke-keys" Oct 30 00:20:05.978668 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:05.978623 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e917f1ed-a050-40ef-b78e-43f1fbdbf461" containerName="revoke-keys" Oct 30 00:20:05.978748 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:05.978739 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e917f1ed-a050-40ef-b78e-43f1fbdbf461" containerName="revoke-keys" Oct 30 00:20:05.982341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:05.982319 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:20:06.017224 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:06.017167 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6"] Oct 30 00:20:06.177701 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:06.177666 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b4143b-d307-4714-bb2c-8704a330753a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-q8gj6\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:20:06.177867 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:06.177707 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-q8gj6\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:20:06.177867 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:06.177796 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-q8gj6\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:20:06.177867 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:06.177829 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-q8gj6\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:20:06.177994 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:06.177886 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-q8gj6\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:20:06.178029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:06.178001 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b4143b-d307-4714-bb2c-8704a330753a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-q8gj6\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:20:06.178155 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:06.178131 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48b4143b-d307-4714-bb2c-8704a330753a-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-q8gj6\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:20:06.178190 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:06.178179 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqjg2\" (UniqueName: \"kubernetes.io/projected/48b4143b-d307-4714-bb2c-8704a330753a-kube-api-access-sqjg2\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-q8gj6\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:20:06.178310 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:06.178297 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-q8gj6\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:20:06.279497 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:06.279461 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-q8gj6\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:20:06.279674 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:06.279518 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b4143b-d307-4714-bb2c-8704a330753a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-q8gj6\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:20:06.279674 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:06.279537 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-q8gj6\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:20:06.279674 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:06.279571 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-q8gj6\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:20:06.279674 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:06.279588 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-q8gj6\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:20:06.279674 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:06.279637 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-q8gj6\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:20:06.279901 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:06.279698 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b4143b-d307-4714-bb2c-8704a330753a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-q8gj6\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:20:06.279901 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:06.279757 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48b4143b-d307-4714-bb2c-8704a330753a-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-q8gj6\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:20:06.279901 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:06.279797 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqjg2\" (UniqueName: \"kubernetes.io/projected/48b4143b-d307-4714-bb2c-8704a330753a-kube-api-access-sqjg2\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-q8gj6\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:20:06.280089 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:06.280048 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-q8gj6\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:20:06.280155 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:06.280095 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-q8gj6\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:20:06.280155 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:06.280128 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-q8gj6\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:20:06.280413 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:06.280390 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-q8gj6\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:20:06.280585 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:06.280556 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b4143b-d307-4714-bb2c-8704a330753a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-q8gj6\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:20:06.281845 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:06.281823 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-q8gj6\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:20:06.282188 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:06.282167 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b4143b-d307-4714-bb2c-8704a330753a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-q8gj6\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:20:06.300296 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:06.300275 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48b4143b-d307-4714-bb2c-8704a330753a-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-q8gj6\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:20:06.300469 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:06.300451 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqjg2\" (UniqueName: \"kubernetes.io/projected/48b4143b-d307-4714-bb2c-8704a330753a-kube-api-access-sqjg2\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-q8gj6\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:20:06.597153 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:06.597055 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:20:06.923383 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:06.923342 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:20:06.923530 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:06.923432 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:20:06.923530 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:06.923478 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:20:06.923603 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:06.923536 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6"] Oct 30 00:20:07.243283 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:07.243169 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" event={"ID":"48b4143b-d307-4714-bb2c-8704a330753a","Type":"ContainerStarted","Data":"0b2e80c59351f503e4626c57b772869beaa0e031ddb5d0c7ffeb8fde9cff2171"} Oct 30 00:20:07.243878 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:07.243839 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" event={"ID":"48b4143b-d307-4714-bb2c-8704a330753a","Type":"ContainerStarted","Data":"22be6ff189790b36c4ac0a28e98f7ec46624884b420163040505927de24bde7d"} Oct 30 00:20:07.305151 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:07.305099 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" podStartSLOduration=2.305086454 podStartE2EDuration="2.305086454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:20:07.294112879 +0000 UTC m=+922.499614083" watchObservedRunningTime="2026-10-30 00:20:07.305086454 +0000 UTC m=+922.510587634" Oct 30 00:20:07.597711 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:07.597678 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:20:07.599483 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:07.599452 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 00:20:07.599624 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:07.599512 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" podUID="48b4143b-d307-4714-bb2c-8704a330753a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 00:20:07.740137 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:07.740094 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e94c9f1-5pg6m"] Oct 30 00:20:07.744592 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:07.744571 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e94c9f1-5pg6m" Oct 30 00:20:07.758041 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:07.758007 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:20:07.758157 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:07.758056 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:20:07.758210 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:07.758181 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:20:07.758251 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:07.758231 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:20:07.763753 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:07.763727 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e94c9f1-5pg6m"] Oct 30 00:20:07.849038 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:07.848955 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ff7382-9b32-4c01-9d13-5df1841b0db6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e94c9f1-5pg6m\" (UID: \"65ff7382-9b32-4c01-9d13-5df1841b0db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e94c9f1-5pg6m" Oct 30 00:20:07.849038 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:07.849010 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk4x4\" (UniqueName: \"kubernetes.io/projected/65ff7382-9b32-4c01-9d13-5df1841b0db6-kube-api-access-zk4x4\") pod \"maas-api-revoke-sub-1e94c9f1-5pg6m\" (UID: \"65ff7382-9b32-4c01-9d13-5df1841b0db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e94c9f1-5pg6m" Oct 30 00:20:07.849238 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:07.849084 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ff7382-9b32-4c01-9d13-5df1841b0db6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e94c9f1-5pg6m\" (UID: \"65ff7382-9b32-4c01-9d13-5df1841b0db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e94c9f1-5pg6m" Oct 30 00:20:07.950679 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:07.950638 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ff7382-9b32-4c01-9d13-5df1841b0db6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e94c9f1-5pg6m\" (UID: \"65ff7382-9b32-4c01-9d13-5df1841b0db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e94c9f1-5pg6m" Oct 30 00:20:07.950906 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:07.950718 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk4x4\" (UniqueName: \"kubernetes.io/projected/65ff7382-9b32-4c01-9d13-5df1841b0db6-kube-api-access-zk4x4\") pod \"maas-api-revoke-sub-1e94c9f1-5pg6m\" (UID: \"65ff7382-9b32-4c01-9d13-5df1841b0db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e94c9f1-5pg6m" Oct 30 00:20:07.950906 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:07.950749 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ff7382-9b32-4c01-9d13-5df1841b0db6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e94c9f1-5pg6m\" (UID: \"65ff7382-9b32-4c01-9d13-5df1841b0db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e94c9f1-5pg6m" Oct 30 00:20:07.951459 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:07.951432 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ff7382-9b32-4c01-9d13-5df1841b0db6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e94c9f1-5pg6m\" (UID: \"65ff7382-9b32-4c01-9d13-5df1841b0db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e94c9f1-5pg6m" Oct 30 00:20:07.953218 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:07.953195 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ff7382-9b32-4c01-9d13-5df1841b0db6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e94c9f1-5pg6m\" (UID: \"65ff7382-9b32-4c01-9d13-5df1841b0db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e94c9f1-5pg6m" Oct 30 00:20:07.971608 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:07.971578 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk4x4\" (UniqueName: \"kubernetes.io/projected/65ff7382-9b32-4c01-9d13-5df1841b0db6-kube-api-access-zk4x4\") pod \"maas-api-revoke-sub-1e94c9f1-5pg6m\" (UID: \"65ff7382-9b32-4c01-9d13-5df1841b0db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e94c9f1-5pg6m" Oct 30 00:20:08.060285 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:08.060251 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e94c9f1-5pg6m" Oct 30 00:20:08.391301 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:08.391273 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e94c9f1-5pg6m"] Oct 30 00:20:08.392272 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:20:08.392206 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ff7382_9b32_4c01_9d13_5df1841b0db6.slice/crio-d697496b75c3d1fd16e331eee0b93ecda0e16a9d12cb0f95df5e72b1a49b464c WatchSource:0}: Error finding container d697496b75c3d1fd16e331eee0b93ecda0e16a9d12cb0f95df5e72b1a49b464c: Status 404 returned error can't find the container with id d697496b75c3d1fd16e331eee0b93ecda0e16a9d12cb0f95df5e72b1a49b464c Oct 30 00:20:08.598433 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:08.598400 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 00:20:08.598615 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:08.598463 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" podUID="48b4143b-d307-4714-bb2c-8704a330753a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 00:20:08.970040 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:08.969959 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv"] Oct 30 00:20:08.974472 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:08.974445 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:20:09.002335 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.002307 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv"] Oct 30 00:20:09.070809 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.070770 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:20:09.071028 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.070821 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l67x5\" (UniqueName: \"kubernetes.io/projected/f84427bf-e708-4616-b20d-417182380594-kube-api-access-l67x5\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:20:09.071028 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.070853 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:20:09.071028 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.070881 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:20:09.071028 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.070942 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f84427bf-e708-4616-b20d-417182380594-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:20:09.071266 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.071182 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:20:09.071328 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.071301 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f84427bf-e708-4616-b20d-417182380594-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:20:09.071385 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.071330 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:20:09.071385 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.071354 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f84427bf-e708-4616-b20d-417182380594-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:20:09.172137 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.172103 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f84427bf-e708-4616-b20d-417182380594-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:20:09.172318 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.172141 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:20:09.172318 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.172165 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f84427bf-e708-4616-b20d-417182380594-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:20:09.172318 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.172188 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:20:09.172318 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.172211 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l67x5\" (UniqueName: \"kubernetes.io/projected/f84427bf-e708-4616-b20d-417182380594-kube-api-access-l67x5\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:20:09.172318 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.172240 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:20:09.172318 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.172259 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:20:09.172318 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.172290 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f84427bf-e708-4616-b20d-417182380594-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:20:09.172568 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.172464 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:20:09.172629 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.172599 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:20:09.172702 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.172681 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:20:09.172866 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.172840 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:20:09.173032 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.172940 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:20:09.173247 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.173226 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f84427bf-e708-4616-b20d-417182380594-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:20:09.174670 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.174643 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:20:09.174879 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.174861 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f84427bf-e708-4616-b20d-417182380594-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:20:09.192856 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.192833 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f84427bf-e708-4616-b20d-417182380594-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:20:09.192975 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.192839 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l67x5\" (UniqueName: \"kubernetes.io/projected/f84427bf-e708-4616-b20d-417182380594-kube-api-access-l67x5\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:20:09.265182 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.265140 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e94c9f1-5pg6m" event={"ID":"65ff7382-9b32-4c01-9d13-5df1841b0db6","Type":"ContainerStarted","Data":"35120da54338ba6c0152436ae97981a7ee9c2a955bd4bcd53d920db4ac4f8ad2"} Oct 30 00:20:09.265355 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.265215 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e94c9f1-5pg6m" event={"ID":"65ff7382-9b32-4c01-9d13-5df1841b0db6","Type":"ContainerStarted","Data":"d697496b75c3d1fd16e331eee0b93ecda0e16a9d12cb0f95df5e72b1a49b464c"} Oct 30 00:20:09.292725 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.292696 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:20:09.381390 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.379972 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e94c9f1-5pg6m" podStartSLOduration=2.379956316 podStartE2EDuration="2.379956316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:20:09.376848007 +0000 UTC m=+924.582349193" watchObservedRunningTime="2026-10-30 00:20:09.379956316 +0000 UTC m=+924.585457497" Oct 30 00:20:09.598126 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.598033 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 00:20:09.598126 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.598101 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" podUID="48b4143b-d307-4714-bb2c-8704a330753a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 00:20:09.644256 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.644229 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv"] Oct 30 00:20:09.662244 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.662208 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:20:09.662365 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.662284 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:20:09.662365 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:09.662315 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:20:10.358958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:10.358923 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" event={"ID":"f84427bf-e708-4616-b20d-417182380594","Type":"ContainerStarted","Data":"af3bb49da4fc6376eadeaa96b3d75d783bef98e419cbfcc5faa13d2c481ea04d"} Oct 30 00:20:10.360946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:10.360871 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" event={"ID":"f84427bf-e708-4616-b20d-417182380594","Type":"ContainerStarted","Data":"853fb4480acfeb024a06839974c6034d5d420c36b860d8363f8409f207c58a5b"} Oct 30 00:20:10.361699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:10.361672 5001 generic.go:356] "Generic (PLEG): container finished" podID="65ff7382-9b32-4c01-9d13-5df1841b0db6" containerID="35120da54338ba6c0152436ae97981a7ee9c2a955bd4bcd53d920db4ac4f8ad2" exitCode=0 Oct 30 00:20:10.361833 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:10.361815 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e94c9f1-5pg6m" event={"ID":"65ff7382-9b32-4c01-9d13-5df1841b0db6","Type":"ContainerDied","Data":"35120da54338ba6c0152436ae97981a7ee9c2a955bd4bcd53d920db4ac4f8ad2"} Oct 30 00:20:10.415937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:10.413500 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" podStartSLOduration=2.413482261 podStartE2EDuration="2.413482261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:20:10.412607091 +0000 UTC m=+925.618108290" watchObservedRunningTime="2026-10-30 00:20:10.413482261 +0000 UTC m=+925.618983459" Oct 30 00:20:10.597515 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:10.597482 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 00:20:10.597778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:10.597749 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" podUID="48b4143b-d307-4714-bb2c-8704a330753a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 00:20:11.294193 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:11.294149 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:20:11.297625 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:11.297594 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 00:20:11.297845 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:11.297822 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" podUID="f84427bf-e708-4616-b20d-417182380594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 00:20:11.598038 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:11.597943 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 00:20:11.598038 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:11.598008 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" podUID="48b4143b-d307-4714-bb2c-8704a330753a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 00:20:11.703098 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:11.703071 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e94c9f1-5pg6m" Oct 30 00:20:11.800250 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:11.800218 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65ff7382-9b32-4c01-9d13-5df1841b0db6-kube-api-access-zk4x4\" (UniqueName: \"kubernetes.io/projected/65ff7382-9b32-4c01-9d13-5df1841b0db6-kube-api-access-zk4x4\") pod \"65ff7382-9b32-4c01-9d13-5df1841b0db6\" (UID: \"65ff7382-9b32-4c01-9d13-5df1841b0db6\") " Oct 30 00:20:11.800416 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:11.800259 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65ff7382-9b32-4c01-9d13-5df1841b0db6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ff7382-9b32-4c01-9d13-5df1841b0db6-maas-api-openshift-ca\") pod \"65ff7382-9b32-4c01-9d13-5df1841b0db6\" (UID: \"65ff7382-9b32-4c01-9d13-5df1841b0db6\") " Oct 30 00:20:11.800416 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:11.800373 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65ff7382-9b32-4c01-9d13-5df1841b0db6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ff7382-9b32-4c01-9d13-5df1841b0db6-maas-api-xks-ca\") pod \"65ff7382-9b32-4c01-9d13-5df1841b0db6\" (UID: \"65ff7382-9b32-4c01-9d13-5df1841b0db6\") " Oct 30 00:20:11.801881 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:11.801831 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65ff7382-9b32-4c01-9d13-5df1841b0db6-maas-api-openshift-ca" pod "65ff7382-9b32-4c01-9d13-5df1841b0db6" (UID: "65ff7382-9b32-4c01-9d13-5df1841b0db6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:20:11.805439 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:11.805399 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65ff7382-9b32-4c01-9d13-5df1841b0db6-maas-api-xks-ca" pod "65ff7382-9b32-4c01-9d13-5df1841b0db6" (UID: "65ff7382-9b32-4c01-9d13-5df1841b0db6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:20:11.867987 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:11.860737 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ff7382-9b32-4c01-9d13-5df1841b0db6-kube-api-access-zk4x4" pod "65ff7382-9b32-4c01-9d13-5df1841b0db6" (UID: "65ff7382-9b32-4c01-9d13-5df1841b0db6"). InnerVolumeSpecName "kube-api-access-zk4x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:20:11.902206 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:11.902164 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk4x4\" (UniqueName: \"kubernetes.io/projected/65ff7382-9b32-4c01-9d13-5df1841b0db6-kube-api-access-zk4x4\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:20:11.902206 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:11.902194 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ff7382-9b32-4c01-9d13-5df1841b0db6-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:20:11.902206 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:11.902205 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ff7382-9b32-4c01-9d13-5df1841b0db6-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:20:12.293883 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:12.293541 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 00:20:12.293883 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:12.293596 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" podUID="f84427bf-e708-4616-b20d-417182380594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 00:20:12.390703 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:12.390661 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e94c9f1-5pg6m" Oct 30 00:20:12.391190 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:12.390765 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e94c9f1-5pg6m" event={"ID":"65ff7382-9b32-4c01-9d13-5df1841b0db6","Type":"ContainerDied","Data":"d697496b75c3d1fd16e331eee0b93ecda0e16a9d12cb0f95df5e72b1a49b464c"} Oct 30 00:20:12.391190 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:12.390796 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d697496b75c3d1fd16e331eee0b93ecda0e16a9d12cb0f95df5e72b1a49b464c" Oct 30 00:20:12.615993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:12.608775 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 00:20:12.615993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:12.608840 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" podUID="48b4143b-d307-4714-bb2c-8704a330753a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 00:20:13.293375 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:13.293336 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 00:20:13.293585 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:13.293399 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" podUID="f84427bf-e708-4616-b20d-417182380594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 00:20:13.598236 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:13.598160 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 00:20:13.598236 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:13.598214 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" podUID="48b4143b-d307-4714-bb2c-8704a330753a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 00:20:14.293521 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:14.293481 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 00:20:14.293724 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:14.293580 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" podUID="f84427bf-e708-4616-b20d-417182380594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 00:20:14.607010 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:14.606397 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 00:20:14.607010 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:14.606473 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" podUID="48b4143b-d307-4714-bb2c-8704a330753a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 00:20:15.299394 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:15.299355 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 00:20:15.299701 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:15.299669 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" podUID="f84427bf-e708-4616-b20d-417182380594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 00:20:15.606103 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:15.604314 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 00:20:15.606103 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:15.604380 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" podUID="48b4143b-d307-4714-bb2c-8704a330753a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 00:20:15.961656 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:15.950161 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-fhf6b"] Oct 30 00:20:15.961656 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:15.951196 5001 cpu_manager.go:414] "containerMap: removing container" podUID="65ff7382-9b32-4c01-9d13-5df1841b0db6" containerName="revoke-keys" Oct 30 00:20:15.961656 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:15.951219 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65ff7382-9b32-4c01-9d13-5df1841b0db6" containerName="revoke-keys" Oct 30 00:20:15.961656 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:15.951389 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="65ff7382-9b32-4c01-9d13-5df1841b0db6" containerName="revoke-keys" Oct 30 00:20:15.981309 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:15.976167 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-fhf6b" Oct 30 00:20:15.991280 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:15.986898 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-fhf6b"] Oct 30 00:20:15.991280 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:15.991035 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-0847e8c0-dockercfg-7g6l5\"" Oct 30 00:20:15.991280 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:15.991107 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-0847e8c0\"" Oct 30 00:20:16.117961 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.108448 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shct4\" (UniqueName: \"kubernetes.io/projected/d7a080fa-80b8-4542-b180-40a65c8bb758-kube-api-access-shct4\") pod \"payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-fhf6b\" (UID: \"d7a080fa-80b8-4542-b180-40a65c8bb758\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-fhf6b" Oct 30 00:20:16.128612 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.118273 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a080fa-80b8-4542-b180-40a65c8bb758-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-fhf6b\" (UID: \"d7a080fa-80b8-4542-b180-40a65c8bb758\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-fhf6b" Oct 30 00:20:16.302935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.302195 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 00:20:16.330936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.313999 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" podUID="f84427bf-e708-4616-b20d-417182380594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 00:20:16.342931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.338262 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shct4\" (UniqueName: \"kubernetes.io/projected/d7a080fa-80b8-4542-b180-40a65c8bb758-kube-api-access-shct4\") pod \"payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-fhf6b\" (UID: \"d7a080fa-80b8-4542-b180-40a65c8bb758\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-fhf6b" Oct 30 00:20:16.342931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.338349 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a080fa-80b8-4542-b180-40a65c8bb758-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-fhf6b\" (UID: \"d7a080fa-80b8-4542-b180-40a65c8bb758\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-fhf6b" Oct 30 00:20:16.342931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.339515 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a080fa-80b8-4542-b180-40a65c8bb758-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-fhf6b\" (UID: \"d7a080fa-80b8-4542-b180-40a65c8bb758\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-fhf6b" Oct 30 00:20:16.394931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.383099 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shct4\" (UniqueName: \"kubernetes.io/projected/d7a080fa-80b8-4542-b180-40a65c8bb758-kube-api-access-shct4\") pod \"payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-fhf6b\" (UID: \"d7a080fa-80b8-4542-b180-40a65c8bb758\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-fhf6b" Oct 30 00:20:16.600801 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.600714 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-fhf6b" Oct 30 00:20:16.661411 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.658960 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7"] Oct 30 00:20:16.677519 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.677486 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7"] Oct 30 00:20:16.677687 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.677541 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp"] Oct 30 00:20:16.685467 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.681822 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:20:16.699386 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.696150 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp"] Oct 30 00:20:16.699386 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.696265 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:20:16.868836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.866864 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/374db7c3-3cfd-4abf-9199-27f55fd2c548-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:20:16.868836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.866943 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:20:16.868836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.866988 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/374db7c3-3cfd-4abf-9199-27f55fd2c548-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:20:16.868836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.867032 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e44a9bb-130f-4749-bd51-ab0fa7958247-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:20:16.868836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.867096 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:20:16.868836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.867123 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e44a9bb-130f-4749-bd51-ab0fa7958247-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:20:16.868836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.867148 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:20:16.868836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.867187 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:20:16.868836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.867289 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:20:16.868836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.867315 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:20:16.868836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.867356 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:20:16.868836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.867398 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:20:16.868836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.867427 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:20:16.868836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.867480 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:20:16.868836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.867531 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e44a9bb-130f-4749-bd51-ab0fa7958247-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:20:16.868836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.867559 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/374db7c3-3cfd-4abf-9199-27f55fd2c548-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:20:16.868836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.867616 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slfrz\" (UniqueName: \"kubernetes.io/projected/374db7c3-3cfd-4abf-9199-27f55fd2c548-kube-api-access-slfrz\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:20:16.868836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.867660 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rg5b\" (UniqueName: \"kubernetes.io/projected/5e44a9bb-130f-4749-bd51-ab0fa7958247-kube-api-access-6rg5b\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:20:16.995418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.976115 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:20:16.995418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.976185 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:20:16.995418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.976233 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e44a9bb-130f-4749-bd51-ab0fa7958247-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:20:16.995418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.976263 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/374db7c3-3cfd-4abf-9199-27f55fd2c548-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:20:16.995418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.976313 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slfrz\" (UniqueName: \"kubernetes.io/projected/374db7c3-3cfd-4abf-9199-27f55fd2c548-kube-api-access-slfrz\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:20:16.995418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.976354 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rg5b\" (UniqueName: \"kubernetes.io/projected/5e44a9bb-130f-4749-bd51-ab0fa7958247-kube-api-access-6rg5b\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:20:16.995418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.976455 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/374db7c3-3cfd-4abf-9199-27f55fd2c548-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:20:16.995418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.976494 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:20:16.995418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.976533 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/374db7c3-3cfd-4abf-9199-27f55fd2c548-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:20:16.995418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.976579 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e44a9bb-130f-4749-bd51-ab0fa7958247-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:20:16.995418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.976659 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:20:16.995418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.976697 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e44a9bb-130f-4749-bd51-ab0fa7958247-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:20:16.995418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.976723 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:20:16.995418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.976761 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:20:16.995418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.976885 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:20:16.995418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.976932 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:20:16.995418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.976979 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:20:16.995418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.977030 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:20:16.995418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.977819 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:20:16.995418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.977943 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:20:16.995418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.978804 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/374db7c3-3cfd-4abf-9199-27f55fd2c548-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:20:16.995418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.979070 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:20:16.995418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.979383 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:20:16.995418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.979488 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e44a9bb-130f-4749-bd51-ab0fa7958247-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:20:16.995418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.979733 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:20:16.995418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.979830 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:20:16.995418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.980116 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:20:16.995418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.981434 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:20:16.995418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.982376 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e44a9bb-130f-4749-bd51-ab0fa7958247-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:20:16.995418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.982637 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:20:16.995418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.993744 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/374db7c3-3cfd-4abf-9199-27f55fd2c548-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:20:16.997825 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:16.997780 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:20:17.100768 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:17.100443 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e44a9bb-130f-4749-bd51-ab0fa7958247-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:20:17.101382 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:17.101307 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slfrz\" (UniqueName: \"kubernetes.io/projected/374db7c3-3cfd-4abf-9199-27f55fd2c548-kube-api-access-slfrz\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:20:17.102022 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:17.101971 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/374db7c3-3cfd-4abf-9199-27f55fd2c548-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:20:17.102138 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:17.102085 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rg5b\" (UniqueName: \"kubernetes.io/projected/5e44a9bb-130f-4749-bd51-ab0fa7958247-kube-api-access-6rg5b\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:20:17.224036 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:17.223949 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-fhf6b"] Oct 30 00:20:17.229409 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:20:17.229375 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a080fa_80b8_4542_b180_40a65c8bb758.slice/crio-b21c9f22e48d58d7dfd9e5bf5d796e169924971ddeb7efc15808ef5d572d30d3 WatchSource:0}: Error finding container b21c9f22e48d58d7dfd9e5bf5d796e169924971ddeb7efc15808ef5d572d30d3: Status 404 returned error can't find the container with id b21c9f22e48d58d7dfd9e5bf5d796e169924971ddeb7efc15808ef5d572d30d3 Oct 30 00:20:17.296757 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:17.296303 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 00:20:17.296757 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:17.296362 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" podUID="f84427bf-e708-4616-b20d-417182380594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 00:20:17.349938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:17.347487 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:20:17.364939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:17.364463 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:20:17.450667 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:20:17.450560 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:20:17.518708 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:17.517596 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-fhf6b" event={"ID":"d7a080fa-80b8-4542-b180-40a65c8bb758","Type":"ContainerStarted","Data":"edb38dd57203af7c4cabdc9e4c5c0e1fb5ff2a7e1b6d2d071b4f1d34e9c7f6bf"} Oct 30 00:20:17.518708 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:17.517662 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-fhf6b" event={"ID":"d7a080fa-80b8-4542-b180-40a65c8bb758","Type":"ContainerStarted","Data":"b21c9f22e48d58d7dfd9e5bf5d796e169924971ddeb7efc15808ef5d572d30d3"} Oct 30 00:20:17.518708 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:17.518642 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-fhf6b" Oct 30 00:20:17.562557 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:17.562492 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-fhf6b" podStartSLOduration=2.562474902 podStartE2EDuration="2.562474902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:20:17.561207989 +0000 UTC m=+932.766709167" watchObservedRunningTime="2026-10-30 00:20:17.562474902 +0000 UTC m=+932.767976085" Oct 30 00:20:17.640094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:17.640063 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:20:17.640256 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:17.640121 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" podUID="48b4143b-d307-4714-bb2c-8704a330753a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:17.853886 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:17.853853 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:20:17.854382 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:17.854352 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:20:17.855946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:17.855882 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:20:17.922354 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:17.920330 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7"] Oct 30 00:20:17.926041 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:17.925446 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp"] Oct 30 00:20:17.926581 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:20:17.926541 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod374db7c3_3cfd_4abf_9199_27f55fd2c548.slice/crio-eb491fcd2da1b40e3a8c26cebf9d42805cefc1524ac0362113becf8a2f63d640 WatchSource:0}: Error finding container eb491fcd2da1b40e3a8c26cebf9d42805cefc1524ac0362113becf8a2f63d640: Status 404 returned error can't find the container with id eb491fcd2da1b40e3a8c26cebf9d42805cefc1524ac0362113becf8a2f63d640 Oct 30 00:20:17.928553 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:20:17.928532 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e44a9bb_130f_4749_bd51_ab0fa7958247.slice/crio-b3854becd30e59666f815f15a5d435219bf7cec36b218df41ecf39cdce0c6096 WatchSource:0}: Error finding container b3854becd30e59666f815f15a5d435219bf7cec36b218df41ecf39cdce0c6096: Status 404 returned error can't find the container with id b3854becd30e59666f815f15a5d435219bf7cec36b218df41ecf39cdce0c6096 Oct 30 00:20:17.930462 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:17.930420 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:20:17.930564 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:17.930543 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:20:17.930607 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:17.930595 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:20:17.931773 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:17.931741 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:20:17.931892 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:17.931816 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:20:17.932005 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:17.931942 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:20:18.293458 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:18.293418 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 00:20:18.293838 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:18.293486 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" podUID="f84427bf-e708-4616-b20d-417182380594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 00:20:18.535935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:18.535628 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" event={"ID":"5e44a9bb-130f-4749-bd51-ab0fa7958247","Type":"ContainerStarted","Data":"aeb62f4e111c193e872f476b470f7547ff79ed11912b31bea78ac5f3dc4e9a91"} Oct 30 00:20:18.535935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:18.535743 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" event={"ID":"5e44a9bb-130f-4749-bd51-ab0fa7958247","Type":"ContainerStarted","Data":"b3854becd30e59666f815f15a5d435219bf7cec36b218df41ecf39cdce0c6096"} Oct 30 00:20:18.542970 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:18.539459 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" event={"ID":"374db7c3-3cfd-4abf-9199-27f55fd2c548","Type":"ContainerStarted","Data":"72658ef052b0c90bdf5341bf06233b2b465936fd985225664e1c22b713934b41"} Oct 30 00:20:18.542970 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:18.539517 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" event={"ID":"374db7c3-3cfd-4abf-9199-27f55fd2c548","Type":"ContainerStarted","Data":"eb491fcd2da1b40e3a8c26cebf9d42805cefc1524ac0362113becf8a2f63d640"} Oct 30 00:20:18.585783 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:18.585658 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" podStartSLOduration=2.585637468 podStartE2EDuration="2.585637468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:20:18.581319437 +0000 UTC m=+933.786820618" watchObservedRunningTime="2026-10-30 00:20:18.585637468 +0000 UTC m=+933.791138652" Oct 30 00:20:18.622560 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:18.622443 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" podStartSLOduration=2.6224232880000002 podStartE2EDuration="2.622423288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:20:18.621200528 +0000 UTC m=+933.826701740" watchObservedRunningTime="2026-10-30 00:20:18.622423288 +0000 UTC m=+933.827924470" Oct 30 00:20:19.298364 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:19.298334 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:20:19.298837 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:19.298811 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:20:19.304921 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:19.299721 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:20:19.368933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:19.348650 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:20:19.368933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:19.350216 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 00:20:19.368933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:19.350268 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" podUID="374db7c3-3cfd-4abf-9199-27f55fd2c548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 00:20:19.372346 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:19.372312 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:20:19.374092 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:19.374045 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 00:20:19.374220 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:19.374121 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" podUID="5e44a9bb-130f-4749-bd51-ab0fa7958247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 00:20:20.358033 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:20.357986 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 00:20:20.358656 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:20.358625 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" podUID="374db7c3-3cfd-4abf-9199-27f55fd2c548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 00:20:20.371664 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:20.371625 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 00:20:20.380660 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:20.380616 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" podUID="5e44a9bb-130f-4749-bd51-ab0fa7958247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 00:20:21.357934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:21.357341 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 00:20:21.357934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:21.357412 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" podUID="374db7c3-3cfd-4abf-9199-27f55fd2c548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 00:20:21.365587 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:21.365350 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 00:20:21.365587 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:21.365411 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" podUID="5e44a9bb-130f-4749-bd51-ab0fa7958247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 00:20:22.351297 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:22.351260 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 00:20:22.351569 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:22.351538 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" podUID="374db7c3-3cfd-4abf-9199-27f55fd2c548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 00:20:22.368941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:22.365398 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 00:20:22.368941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:22.365458 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" podUID="5e44a9bb-130f-4749-bd51-ab0fa7958247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 00:20:23.162952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.156977 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b76cbf84-8p4mk"] Oct 30 00:20:23.173875 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.173849 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76cbf84-8p4mk" Oct 30 00:20:23.176957 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.176151 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b76cbf84-8p4mk"] Oct 30 00:20:23.187539 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.186540 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:20:23.196441 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.196418 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:20:23.196522 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.196480 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:20:23.196522 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.196508 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:20:23.302044 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.302006 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2nff\" (UniqueName: \"kubernetes.io/projected/6f69d870-15fa-4208-abca-2bfa331060a1-kube-api-access-k2nff\") pod \"maas-api-revoke-sub-b76cbf84-8p4mk\" (UID: \"6f69d870-15fa-4208-abca-2bfa331060a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76cbf84-8p4mk" Oct 30 00:20:23.302323 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.302293 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f69d870-15fa-4208-abca-2bfa331060a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b76cbf84-8p4mk\" (UID: \"6f69d870-15fa-4208-abca-2bfa331060a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76cbf84-8p4mk" Oct 30 00:20:23.302416 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.302388 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f69d870-15fa-4208-abca-2bfa331060a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b76cbf84-8p4mk\" (UID: \"6f69d870-15fa-4208-abca-2bfa331060a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76cbf84-8p4mk" Oct 30 00:20:23.374114 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.357815 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 00:20:23.374777 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.374739 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" podUID="374db7c3-3cfd-4abf-9199-27f55fd2c548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 00:20:23.379694 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.379650 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 00:20:23.379921 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.379883 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" podUID="5e44a9bb-130f-4749-bd51-ab0fa7958247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 00:20:23.404097 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.404070 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f69d870-15fa-4208-abca-2bfa331060a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b76cbf84-8p4mk\" (UID: \"6f69d870-15fa-4208-abca-2bfa331060a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76cbf84-8p4mk" Oct 30 00:20:23.404291 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.404273 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f69d870-15fa-4208-abca-2bfa331060a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b76cbf84-8p4mk\" (UID: \"6f69d870-15fa-4208-abca-2bfa331060a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76cbf84-8p4mk" Oct 30 00:20:23.404498 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.404478 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2nff\" (UniqueName: \"kubernetes.io/projected/6f69d870-15fa-4208-abca-2bfa331060a1-kube-api-access-k2nff\") pod \"maas-api-revoke-sub-b76cbf84-8p4mk\" (UID: \"6f69d870-15fa-4208-abca-2bfa331060a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76cbf84-8p4mk" Oct 30 00:20:23.405017 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.404995 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f69d870-15fa-4208-abca-2bfa331060a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b76cbf84-8p4mk\" (UID: \"6f69d870-15fa-4208-abca-2bfa331060a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76cbf84-8p4mk" Oct 30 00:20:23.458634 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.458547 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f69d870-15fa-4208-abca-2bfa331060a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b76cbf84-8p4mk\" (UID: \"6f69d870-15fa-4208-abca-2bfa331060a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76cbf84-8p4mk" Oct 30 00:20:23.461600 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.461568 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2nff\" (UniqueName: \"kubernetes.io/projected/6f69d870-15fa-4208-abca-2bfa331060a1-kube-api-access-k2nff\") pod \"maas-api-revoke-sub-b76cbf84-8p4mk\" (UID: \"6f69d870-15fa-4208-abca-2bfa331060a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76cbf84-8p4mk" Oct 30 00:20:23.464983 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.464955 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2dd5d03-zgjzc"] Oct 30 00:20:23.475188 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.474248 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2dd5d03-zgjzc" Oct 30 00:20:23.479639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.479611 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2dd5d03-zgjzc"] Oct 30 00:20:23.491902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.491881 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76cbf84-8p4mk" Oct 30 00:20:23.506331 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.506290 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9fwj\" (UniqueName: \"kubernetes.io/projected/304c6e56-6ae0-4f85-b8f7-cad3069bcf50-kube-api-access-m9fwj\") pod \"maas-api-revoke-sub-c2dd5d03-zgjzc\" (UID: \"304c6e56-6ae0-4f85-b8f7-cad3069bcf50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2dd5d03-zgjzc" Oct 30 00:20:23.506465 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.506354 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/304c6e56-6ae0-4f85-b8f7-cad3069bcf50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2dd5d03-zgjzc\" (UID: \"304c6e56-6ae0-4f85-b8f7-cad3069bcf50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2dd5d03-zgjzc" Oct 30 00:20:23.506859 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.506561 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/304c6e56-6ae0-4f85-b8f7-cad3069bcf50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2dd5d03-zgjzc\" (UID: \"304c6e56-6ae0-4f85-b8f7-cad3069bcf50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2dd5d03-zgjzc" Oct 30 00:20:23.611938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.608384 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9fwj\" (UniqueName: \"kubernetes.io/projected/304c6e56-6ae0-4f85-b8f7-cad3069bcf50-kube-api-access-m9fwj\") pod \"maas-api-revoke-sub-c2dd5d03-zgjzc\" (UID: \"304c6e56-6ae0-4f85-b8f7-cad3069bcf50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2dd5d03-zgjzc" Oct 30 00:20:23.611938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.608444 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/304c6e56-6ae0-4f85-b8f7-cad3069bcf50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2dd5d03-zgjzc\" (UID: \"304c6e56-6ae0-4f85-b8f7-cad3069bcf50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2dd5d03-zgjzc" Oct 30 00:20:23.611938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.608576 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/304c6e56-6ae0-4f85-b8f7-cad3069bcf50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2dd5d03-zgjzc\" (UID: \"304c6e56-6ae0-4f85-b8f7-cad3069bcf50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2dd5d03-zgjzc" Oct 30 00:20:23.629937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.620141 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/304c6e56-6ae0-4f85-b8f7-cad3069bcf50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2dd5d03-zgjzc\" (UID: \"304c6e56-6ae0-4f85-b8f7-cad3069bcf50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2dd5d03-zgjzc" Oct 30 00:20:23.651936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.632266 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/304c6e56-6ae0-4f85-b8f7-cad3069bcf50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2dd5d03-zgjzc\" (UID: \"304c6e56-6ae0-4f85-b8f7-cad3069bcf50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2dd5d03-zgjzc" Oct 30 00:20:23.658933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.654976 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9fwj\" (UniqueName: \"kubernetes.io/projected/304c6e56-6ae0-4f85-b8f7-cad3069bcf50-kube-api-access-m9fwj\") pod \"maas-api-revoke-sub-c2dd5d03-zgjzc\" (UID: \"304c6e56-6ae0-4f85-b8f7-cad3069bcf50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2dd5d03-zgjzc" Oct 30 00:20:23.797738 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.797707 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2dd5d03-zgjzc" Oct 30 00:20:23.993078 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:23.992809 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b76cbf84-8p4mk"] Oct 30 00:20:24.369594 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:24.367259 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 00:20:24.369594 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:24.367307 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" podUID="5e44a9bb-130f-4749-bd51-ab0fa7958247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 00:20:24.370475 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:24.369821 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 00:20:24.370475 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:24.369861 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" podUID="374db7c3-3cfd-4abf-9199-27f55fd2c548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 00:20:24.377040 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:24.370985 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2dd5d03-zgjzc"] Oct 30 00:20:24.763861 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:24.763825 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2dd5d03-zgjzc" event={"ID":"304c6e56-6ae0-4f85-b8f7-cad3069bcf50","Type":"ContainerStarted","Data":"d1334c36ec4282fa691c738c119f0ff47b5516f3fc9b1bc841ddc05a333bd204"} Oct 30 00:20:24.765980 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:24.765949 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76cbf84-8p4mk" event={"ID":"6f69d870-15fa-4208-abca-2bfa331060a1","Type":"ContainerStarted","Data":"11b033b2d8b2d3fa4ff82d0a7858e42f5002a34c34490bbe6a38b0090eeeeea4"} Oct 30 00:20:24.766135 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:24.766007 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76cbf84-8p4mk" event={"ID":"6f69d870-15fa-4208-abca-2bfa331060a1","Type":"ContainerStarted","Data":"cab1d6ccc5f0400c324f973ae6c4775502c286cbff2ffe74194778fde754fc78"} Oct 30 00:20:24.802760 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:24.802708 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76cbf84-8p4mk" podStartSLOduration=1.8026946000000001 podStartE2EDuration="1.8026946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:20:24.800106315 +0000 UTC m=+940.005607496" watchObservedRunningTime="2026-10-30 00:20:24.8026946 +0000 UTC m=+940.008195780" Oct 30 00:20:25.348894 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:25.348854 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 00:20:25.349077 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:25.348942 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" podUID="374db7c3-3cfd-4abf-9199-27f55fd2c548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 00:20:25.374389 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:25.373065 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 00:20:25.374389 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:25.373125 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" podUID="5e44a9bb-130f-4749-bd51-ab0fa7958247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 00:20:25.893625 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:25.893530 5001 generic.go:356] "Generic (PLEG): container finished" podID="304c6e56-6ae0-4f85-b8f7-cad3069bcf50" containerID="eb10c274bdbdf37c4c8c47cbff0c1eb4ac02a310ba576905d1483d2cd0ac0b15" exitCode=0 Oct 30 00:20:25.894346 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:25.894316 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2dd5d03-zgjzc" event={"ID":"304c6e56-6ae0-4f85-b8f7-cad3069bcf50","Type":"ContainerDied","Data":"eb10c274bdbdf37c4c8c47cbff0c1eb4ac02a310ba576905d1483d2cd0ac0b15"} Oct 30 00:20:25.897827 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:25.897747 5001 generic.go:356] "Generic (PLEG): container finished" podID="6f69d870-15fa-4208-abca-2bfa331060a1" containerID="11b033b2d8b2d3fa4ff82d0a7858e42f5002a34c34490bbe6a38b0090eeeeea4" exitCode=0 Oct 30 00:20:25.897952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:25.897879 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76cbf84-8p4mk" event={"ID":"6f69d870-15fa-4208-abca-2bfa331060a1","Type":"ContainerDied","Data":"11b033b2d8b2d3fa4ff82d0a7858e42f5002a34c34490bbe6a38b0090eeeeea4"} Oct 30 00:20:26.353696 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:26.353666 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:20:26.354130 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:26.354105 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:20:26.354757 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:26.354739 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:20:26.371039 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:26.371018 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:20:26.371294 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:26.371280 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:20:26.371984 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:26.371968 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:20:28.595110 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:28.595079 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2dd5d03-zgjzc" Oct 30 00:20:28.647457 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:28.645039 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:28.648345 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:28.646834 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76cbf84-8p4mk" Oct 30 00:20:28.749254 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:28.749206 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f69d870-15fa-4208-abca-2bfa331060a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f69d870-15fa-4208-abca-2bfa331060a1-maas-api-openshift-ca\") pod \"6f69d870-15fa-4208-abca-2bfa331060a1\" (UID: \"6f69d870-15fa-4208-abca-2bfa331060a1\") " Oct 30 00:20:28.763796 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:28.752714 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f69d870-15fa-4208-abca-2bfa331060a1-kube-api-access-k2nff\" (UniqueName: \"kubernetes.io/projected/6f69d870-15fa-4208-abca-2bfa331060a1-kube-api-access-k2nff\") pod \"6f69d870-15fa-4208-abca-2bfa331060a1\" (UID: \"6f69d870-15fa-4208-abca-2bfa331060a1\") " Oct 30 00:20:28.763796 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:28.752847 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f69d870-15fa-4208-abca-2bfa331060a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f69d870-15fa-4208-abca-2bfa331060a1-maas-api-xks-ca\") pod \"6f69d870-15fa-4208-abca-2bfa331060a1\" (UID: \"6f69d870-15fa-4208-abca-2bfa331060a1\") " Oct 30 00:20:28.763796 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:28.752967 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/304c6e56-6ae0-4f85-b8f7-cad3069bcf50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/304c6e56-6ae0-4f85-b8f7-cad3069bcf50-maas-api-xks-ca\") pod \"304c6e56-6ae0-4f85-b8f7-cad3069bcf50\" (UID: \"304c6e56-6ae0-4f85-b8f7-cad3069bcf50\") " Oct 30 00:20:28.765367 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:28.749847 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f69d870-15fa-4208-abca-2bfa331060a1-maas-api-openshift-ca" pod "6f69d870-15fa-4208-abca-2bfa331060a1" (UID: "6f69d870-15fa-4208-abca-2bfa331060a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:20:28.841968 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:28.841697 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/304c6e56-6ae0-4f85-b8f7-cad3069bcf50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/304c6e56-6ae0-4f85-b8f7-cad3069bcf50-maas-api-openshift-ca\") pod \"304c6e56-6ae0-4f85-b8f7-cad3069bcf50\" (UID: \"304c6e56-6ae0-4f85-b8f7-cad3069bcf50\") " Oct 30 00:20:28.841968 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:28.841803 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/304c6e56-6ae0-4f85-b8f7-cad3069bcf50-kube-api-access-m9fwj\" (UniqueName: \"kubernetes.io/projected/304c6e56-6ae0-4f85-b8f7-cad3069bcf50-kube-api-access-m9fwj\") pod \"304c6e56-6ae0-4f85-b8f7-cad3069bcf50\" (UID: \"304c6e56-6ae0-4f85-b8f7-cad3069bcf50\") " Oct 30 00:20:28.871897 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:28.867333 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/304c6e56-6ae0-4f85-b8f7-cad3069bcf50-maas-api-openshift-ca" pod "304c6e56-6ae0-4f85-b8f7-cad3069bcf50" (UID: "304c6e56-6ae0-4f85-b8f7-cad3069bcf50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:20:28.871897 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:28.868361 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f69d870-15fa-4208-abca-2bfa331060a1-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:20:28.947117 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:28.947049 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f69d870-15fa-4208-abca-2bfa331060a1-kube-api-access-k2nff" pod "6f69d870-15fa-4208-abca-2bfa331060a1" (UID: "6f69d870-15fa-4208-abca-2bfa331060a1"). InnerVolumeSpecName "kube-api-access-k2nff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:20:28.949705 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:28.949657 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f69d870-15fa-4208-abca-2bfa331060a1-maas-api-xks-ca" pod "6f69d870-15fa-4208-abca-2bfa331060a1" (UID: "6f69d870-15fa-4208-abca-2bfa331060a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:20:28.952599 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:28.951939 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/304c6e56-6ae0-4f85-b8f7-cad3069bcf50-maas-api-xks-ca" pod "304c6e56-6ae0-4f85-b8f7-cad3069bcf50" (UID: "304c6e56-6ae0-4f85-b8f7-cad3069bcf50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:20:28.952878 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:28.952853 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/304c6e56-6ae0-4f85-b8f7-cad3069bcf50-kube-api-access-m9fwj" pod "304c6e56-6ae0-4f85-b8f7-cad3069bcf50" (UID: "304c6e56-6ae0-4f85-b8f7-cad3069bcf50"). InnerVolumeSpecName "kube-api-access-m9fwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:20:29.051225 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:29.051192 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f69d870-15fa-4208-abca-2bfa331060a1-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:20:29.064667 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:29.064640 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/304c6e56-6ae0-4f85-b8f7-cad3069bcf50-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:20:29.064886 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:29.064869 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/304c6e56-6ae0-4f85-b8f7-cad3069bcf50-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:20:29.065169 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:29.065150 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9fwj\" (UniqueName: \"kubernetes.io/projected/304c6e56-6ae0-4f85-b8f7-cad3069bcf50-kube-api-access-m9fwj\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:20:29.065286 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:29.065271 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2nff\" (UniqueName: \"kubernetes.io/projected/6f69d870-15fa-4208-abca-2bfa331060a1-kube-api-access-k2nff\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:20:29.065376 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:29.052542 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:29.108594 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:29.107651 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:29.326636 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:29.325313 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 00:20:29.356124 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:20:29.355867 5001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f69d870_15fa_4208_abca_2bfa331060a1.slice/crio-cab1d6ccc5f0400c324f973ae6c4775502c286cbff2ffe74194778fde754fc78\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod304c6e56_6ae0_4f85_b8f7_cad3069bcf50.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod304c6e56_6ae0_4f85_b8f7_cad3069bcf50.slice/crio-d1334c36ec4282fa691c738c119f0ff47b5516f3fc9b1bc841ddc05a333bd204\": RecentStats: unable to find data in memory cache]" Oct 30 00:20:29.371362 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:29.365423 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2dd5d03-zgjzc" event={"ID":"304c6e56-6ae0-4f85-b8f7-cad3069bcf50","Type":"ContainerDied","Data":"d1334c36ec4282fa691c738c119f0ff47b5516f3fc9b1bc841ddc05a333bd204"} Oct 30 00:20:29.371362 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:29.365482 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1334c36ec4282fa691c738c119f0ff47b5516f3fc9b1bc841ddc05a333bd204" Oct 30 00:20:29.371362 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:29.365565 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2dd5d03-zgjzc" Oct 30 00:20:29.383154 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:29.381986 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76cbf84-8p4mk" event={"ID":"6f69d870-15fa-4208-abca-2bfa331060a1","Type":"ContainerDied","Data":"cab1d6ccc5f0400c324f973ae6c4775502c286cbff2ffe74194778fde754fc78"} Oct 30 00:20:29.383154 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:29.382043 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cab1d6ccc5f0400c324f973ae6c4775502c286cbff2ffe74194778fde754fc78" Oct 30 00:20:29.383154 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:29.382116 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76cbf84-8p4mk" Oct 30 00:20:29.640192 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:29.640146 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-fhf6b" Oct 30 00:20:29.838120 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:29.838067 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:20:29.838430 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:29.838400 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" podUID="38fbc17b-0fb6-46c8-83af-53b83fa2bcb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:32.512561 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:32.512523 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53c6eb88-thnxg"] Oct 30 00:20:32.513521 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:32.513484 5001 cpu_manager.go:414] "containerMap: removing container" podUID="304c6e56-6ae0-4f85-b8f7-cad3069bcf50" containerName="revoke-keys" Oct 30 00:20:32.513521 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:32.513510 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="304c6e56-6ae0-4f85-b8f7-cad3069bcf50" containerName="revoke-keys" Oct 30 00:20:32.513657 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:32.513529 5001 cpu_manager.go:414] "containerMap: removing container" podUID="6f69d870-15fa-4208-abca-2bfa331060a1" containerName="revoke-keys" Oct 30 00:20:32.513657 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:32.513538 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f69d870-15fa-4208-abca-2bfa331060a1" containerName="revoke-keys" Oct 30 00:20:32.513777 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:32.513764 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="304c6e56-6ae0-4f85-b8f7-cad3069bcf50" containerName="revoke-keys" Oct 30 00:20:32.513815 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:32.513781 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f69d870-15fa-4208-abca-2bfa331060a1" containerName="revoke-keys" Oct 30 00:20:32.518247 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:32.518229 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6eb88-thnxg" Oct 30 00:20:32.534592 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:32.534563 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53c6eb88-thnxg"] Oct 30 00:20:32.543805 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:32.543777 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:20:32.543907 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:32.543866 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:20:32.543987 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:32.543944 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:20:32.544021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:32.543985 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:20:32.689675 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:32.689637 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7490cfd5-895e-4d7b-a842-02bf605da03f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53c6eb88-thnxg\" (UID: \"7490cfd5-895e-4d7b-a842-02bf605da03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6eb88-thnxg" Oct 30 00:20:32.689854 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:32.689805 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7490cfd5-895e-4d7b-a842-02bf605da03f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53c6eb88-thnxg\" (UID: \"7490cfd5-895e-4d7b-a842-02bf605da03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6eb88-thnxg" Oct 30 00:20:32.690037 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:32.690018 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndppn\" (UniqueName: \"kubernetes.io/projected/7490cfd5-895e-4d7b-a842-02bf605da03f-kube-api-access-ndppn\") pod \"maas-api-revoke-sub-53c6eb88-thnxg\" (UID: \"7490cfd5-895e-4d7b-a842-02bf605da03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6eb88-thnxg" Oct 30 00:20:32.803987 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:32.797978 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7490cfd5-895e-4d7b-a842-02bf605da03f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53c6eb88-thnxg\" (UID: \"7490cfd5-895e-4d7b-a842-02bf605da03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6eb88-thnxg" Oct 30 00:20:32.803987 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:32.798195 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndppn\" (UniqueName: \"kubernetes.io/projected/7490cfd5-895e-4d7b-a842-02bf605da03f-kube-api-access-ndppn\") pod \"maas-api-revoke-sub-53c6eb88-thnxg\" (UID: \"7490cfd5-895e-4d7b-a842-02bf605da03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6eb88-thnxg" Oct 30 00:20:32.803987 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:32.798328 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7490cfd5-895e-4d7b-a842-02bf605da03f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53c6eb88-thnxg\" (UID: \"7490cfd5-895e-4d7b-a842-02bf605da03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6eb88-thnxg" Oct 30 00:20:32.803987 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:32.799366 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7490cfd5-895e-4d7b-a842-02bf605da03f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53c6eb88-thnxg\" (UID: \"7490cfd5-895e-4d7b-a842-02bf605da03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6eb88-thnxg" Oct 30 00:20:32.819162 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:32.819124 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7490cfd5-895e-4d7b-a842-02bf605da03f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53c6eb88-thnxg\" (UID: \"7490cfd5-895e-4d7b-a842-02bf605da03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6eb88-thnxg" Oct 30 00:20:32.822198 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:32.821539 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndppn\" (UniqueName: \"kubernetes.io/projected/7490cfd5-895e-4d7b-a842-02bf605da03f-kube-api-access-ndppn\") pod \"maas-api-revoke-sub-53c6eb88-thnxg\" (UID: \"7490cfd5-895e-4d7b-a842-02bf605da03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6eb88-thnxg" Oct 30 00:20:32.845932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:32.837100 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6eb88-thnxg" Oct 30 00:20:33.154254 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:33.154221 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53c6eb88-thnxg"] Oct 30 00:20:33.155346 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:20:33.155318 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7490cfd5_895e_4d7b_a842_02bf605da03f.slice/crio-39d1183736941fcb07f0e4fed171907899a8ed3462b0bbdff7078dc1c9ab1146 WatchSource:0}: Error finding container 39d1183736941fcb07f0e4fed171907899a8ed3462b0bbdff7078dc1c9ab1146: Status 404 returned error can't find the container with id 39d1183736941fcb07f0e4fed171907899a8ed3462b0bbdff7078dc1c9ab1146 Oct 30 00:20:33.420126 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:33.420041 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6eb88-thnxg" event={"ID":"7490cfd5-895e-4d7b-a842-02bf605da03f","Type":"ContainerStarted","Data":"39d1183736941fcb07f0e4fed171907899a8ed3462b0bbdff7078dc1c9ab1146"} Oct 30 00:20:33.494732 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:33.494697 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95fe8ad5-qjctz"] Oct 30 00:20:33.500274 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:33.500250 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe8ad5-qjctz" Oct 30 00:20:33.519653 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:33.519331 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95fe8ad5-qjctz"] Oct 30 00:20:33.609473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:33.609440 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sghrn\" (UniqueName: \"kubernetes.io/projected/e23c62e9-023c-4ff6-9c35-b724da9a824d-kube-api-access-sghrn\") pod \"maas-api-revoke-sub-95fe8ad5-qjctz\" (UID: \"e23c62e9-023c-4ff6-9c35-b724da9a824d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe8ad5-qjctz" Oct 30 00:20:33.609473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:33.609476 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e23c62e9-023c-4ff6-9c35-b724da9a824d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95fe8ad5-qjctz\" (UID: \"e23c62e9-023c-4ff6-9c35-b724da9a824d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe8ad5-qjctz" Oct 30 00:20:33.609723 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:33.609710 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e23c62e9-023c-4ff6-9c35-b724da9a824d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95fe8ad5-qjctz\" (UID: \"e23c62e9-023c-4ff6-9c35-b724da9a824d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe8ad5-qjctz" Oct 30 00:20:33.710619 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:33.710535 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sghrn\" (UniqueName: \"kubernetes.io/projected/e23c62e9-023c-4ff6-9c35-b724da9a824d-kube-api-access-sghrn\") pod \"maas-api-revoke-sub-95fe8ad5-qjctz\" (UID: \"e23c62e9-023c-4ff6-9c35-b724da9a824d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe8ad5-qjctz" Oct 30 00:20:33.710619 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:33.710571 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e23c62e9-023c-4ff6-9c35-b724da9a824d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95fe8ad5-qjctz\" (UID: \"e23c62e9-023c-4ff6-9c35-b724da9a824d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe8ad5-qjctz" Oct 30 00:20:33.710794 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:33.710757 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e23c62e9-023c-4ff6-9c35-b724da9a824d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95fe8ad5-qjctz\" (UID: \"e23c62e9-023c-4ff6-9c35-b724da9a824d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe8ad5-qjctz" Oct 30 00:20:33.711447 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:33.711423 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e23c62e9-023c-4ff6-9c35-b724da9a824d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95fe8ad5-qjctz\" (UID: \"e23c62e9-023c-4ff6-9c35-b724da9a824d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe8ad5-qjctz" Oct 30 00:20:33.712939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:33.712924 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e23c62e9-023c-4ff6-9c35-b724da9a824d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95fe8ad5-qjctz\" (UID: \"e23c62e9-023c-4ff6-9c35-b724da9a824d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe8ad5-qjctz" Oct 30 00:20:33.733927 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:33.733889 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sghrn\" (UniqueName: \"kubernetes.io/projected/e23c62e9-023c-4ff6-9c35-b724da9a824d-kube-api-access-sghrn\") pod \"maas-api-revoke-sub-95fe8ad5-qjctz\" (UID: \"e23c62e9-023c-4ff6-9c35-b724da9a824d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe8ad5-qjctz" Oct 30 00:20:33.825676 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:33.825646 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe8ad5-qjctz" Oct 30 00:20:33.828097 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:33.828072 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-914a49ab-x96qs"] Oct 30 00:20:33.832520 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:33.832501 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-914a49ab-x96qs" Oct 30 00:20:33.853527 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:33.853499 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-914a49ab-x96qs"] Oct 30 00:20:34.014394 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:34.014363 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rth9m\" (UniqueName: \"kubernetes.io/projected/cad7150c-93d8-4200-abbb-a50f1d7449a2-kube-api-access-rth9m\") pod \"maas-api-revoke-sub-914a49ab-x96qs\" (UID: \"cad7150c-93d8-4200-abbb-a50f1d7449a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-914a49ab-x96qs" Oct 30 00:20:34.014559 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:34.014411 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cad7150c-93d8-4200-abbb-a50f1d7449a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-914a49ab-x96qs\" (UID: \"cad7150c-93d8-4200-abbb-a50f1d7449a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-914a49ab-x96qs" Oct 30 00:20:34.014559 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:34.014526 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cad7150c-93d8-4200-abbb-a50f1d7449a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-914a49ab-x96qs\" (UID: \"cad7150c-93d8-4200-abbb-a50f1d7449a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-914a49ab-x96qs" Oct 30 00:20:34.122024 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:34.117198 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rth9m\" (UniqueName: \"kubernetes.io/projected/cad7150c-93d8-4200-abbb-a50f1d7449a2-kube-api-access-rth9m\") pod \"maas-api-revoke-sub-914a49ab-x96qs\" (UID: \"cad7150c-93d8-4200-abbb-a50f1d7449a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-914a49ab-x96qs" Oct 30 00:20:34.122024 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:34.117264 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cad7150c-93d8-4200-abbb-a50f1d7449a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-914a49ab-x96qs\" (UID: \"cad7150c-93d8-4200-abbb-a50f1d7449a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-914a49ab-x96qs" Oct 30 00:20:34.122024 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:34.117316 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cad7150c-93d8-4200-abbb-a50f1d7449a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-914a49ab-x96qs\" (UID: \"cad7150c-93d8-4200-abbb-a50f1d7449a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-914a49ab-x96qs" Oct 30 00:20:34.133931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:34.132517 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cad7150c-93d8-4200-abbb-a50f1d7449a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-914a49ab-x96qs\" (UID: \"cad7150c-93d8-4200-abbb-a50f1d7449a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-914a49ab-x96qs" Oct 30 00:20:34.193933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:34.193656 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cad7150c-93d8-4200-abbb-a50f1d7449a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-914a49ab-x96qs\" (UID: \"cad7150c-93d8-4200-abbb-a50f1d7449a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-914a49ab-x96qs" Oct 30 00:20:34.201934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:34.194451 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rth9m\" (UniqueName: \"kubernetes.io/projected/cad7150c-93d8-4200-abbb-a50f1d7449a2-kube-api-access-rth9m\") pod \"maas-api-revoke-sub-914a49ab-x96qs\" (UID: \"cad7150c-93d8-4200-abbb-a50f1d7449a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-914a49ab-x96qs" Oct 30 00:20:34.376022 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:34.375980 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95fe8ad5-qjctz"] Oct 30 00:20:34.447631 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:34.447592 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe8ad5-qjctz" event={"ID":"e23c62e9-023c-4ff6-9c35-b724da9a824d","Type":"ContainerStarted","Data":"d851e17d0741761e30a6a05b093dd246c4a88841ba1a6cf7e87600ef673cca8b"} Oct 30 00:20:34.450219 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:34.450179 5001 generic.go:356] "Generic (PLEG): container finished" podID="7490cfd5-895e-4d7b-a842-02bf605da03f" containerID="4017a53718d7ee53742f39aed265005721fcc99d076d9036815ca7451e243975" exitCode=0 Oct 30 00:20:34.450485 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:34.450319 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6eb88-thnxg" event={"ID":"7490cfd5-895e-4d7b-a842-02bf605da03f","Type":"ContainerDied","Data":"4017a53718d7ee53742f39aed265005721fcc99d076d9036815ca7451e243975"} Oct 30 00:20:34.485155 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:34.484733 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-914a49ab-x96qs" Oct 30 00:20:34.850728 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:34.850700 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-914a49ab-x96qs"] Oct 30 00:20:34.910657 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:20:34.910591 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad7150c_93d8_4200_abbb_a50f1d7449a2.slice/crio-3af359ade1f8891407ebba2a14e801b705be4a2919dba8b3c969cd567f24e8ae WatchSource:0}: Error finding container 3af359ade1f8891407ebba2a14e801b705be4a2919dba8b3c969cd567f24e8ae: Status 404 returned error can't find the container with id 3af359ade1f8891407ebba2a14e801b705be4a2919dba8b3c969cd567f24e8ae Oct 30 00:20:35.460071 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:35.460041 5001 generic.go:356] "Generic (PLEG): container finished" podID="e23c62e9-023c-4ff6-9c35-b724da9a824d" containerID="0cb53eeae4bd1e5ed7506b877b0d496b6c68857efd850524a4395afa1bba4f30" exitCode=0 Oct 30 00:20:35.460192 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:35.460117 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe8ad5-qjctz" event={"ID":"e23c62e9-023c-4ff6-9c35-b724da9a824d","Type":"ContainerDied","Data":"0cb53eeae4bd1e5ed7506b877b0d496b6c68857efd850524a4395afa1bba4f30"} Oct 30 00:20:35.461775 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:35.461750 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-914a49ab-x96qs" event={"ID":"cad7150c-93d8-4200-abbb-a50f1d7449a2","Type":"ContainerStarted","Data":"ddf487abfd86f974827726f1c8f710c14f4fb25ffd16efab8fc46f4fc756a1e3"} Oct 30 00:20:35.461876 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:35.461795 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-914a49ab-x96qs" event={"ID":"cad7150c-93d8-4200-abbb-a50f1d7449a2","Type":"ContainerStarted","Data":"3af359ade1f8891407ebba2a14e801b705be4a2919dba8b3c969cd567f24e8ae"} Oct 30 00:20:35.542598 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:35.542536 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-914a49ab-x96qs" podStartSLOduration=2.542511277 podStartE2EDuration="2.542511277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:20:35.536424542 +0000 UTC m=+950.741925723" watchObservedRunningTime="2026-10-30 00:20:35.542511277 +0000 UTC m=+950.748012459" Oct 30 00:20:35.640518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:35.640495 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6eb88-thnxg" Oct 30 00:20:35.766506 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:35.766482 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7490cfd5-895e-4d7b-a842-02bf605da03f-kube-api-access-ndppn\" (UniqueName: \"kubernetes.io/projected/7490cfd5-895e-4d7b-a842-02bf605da03f-kube-api-access-ndppn\") pod \"7490cfd5-895e-4d7b-a842-02bf605da03f\" (UID: \"7490cfd5-895e-4d7b-a842-02bf605da03f\") " Oct 30 00:20:35.766655 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:35.766524 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7490cfd5-895e-4d7b-a842-02bf605da03f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7490cfd5-895e-4d7b-a842-02bf605da03f-maas-api-openshift-ca\") pod \"7490cfd5-895e-4d7b-a842-02bf605da03f\" (UID: \"7490cfd5-895e-4d7b-a842-02bf605da03f\") " Oct 30 00:20:35.766655 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:35.766624 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7490cfd5-895e-4d7b-a842-02bf605da03f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7490cfd5-895e-4d7b-a842-02bf605da03f-maas-api-xks-ca\") pod \"7490cfd5-895e-4d7b-a842-02bf605da03f\" (UID: \"7490cfd5-895e-4d7b-a842-02bf605da03f\") " Oct 30 00:20:35.767017 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:35.766982 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7490cfd5-895e-4d7b-a842-02bf605da03f-maas-api-openshift-ca" pod "7490cfd5-895e-4d7b-a842-02bf605da03f" (UID: "7490cfd5-895e-4d7b-a842-02bf605da03f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:20:35.767130 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:35.767107 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7490cfd5-895e-4d7b-a842-02bf605da03f-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:20:35.768585 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:35.768558 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7490cfd5-895e-4d7b-a842-02bf605da03f-maas-api-xks-ca" pod "7490cfd5-895e-4d7b-a842-02bf605da03f" (UID: "7490cfd5-895e-4d7b-a842-02bf605da03f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:20:35.768678 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:35.768607 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7490cfd5-895e-4d7b-a842-02bf605da03f-kube-api-access-ndppn" pod "7490cfd5-895e-4d7b-a842-02bf605da03f" (UID: "7490cfd5-895e-4d7b-a842-02bf605da03f"). InnerVolumeSpecName "kube-api-access-ndppn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:20:35.868764 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:35.868734 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndppn\" (UniqueName: \"kubernetes.io/projected/7490cfd5-895e-4d7b-a842-02bf605da03f-kube-api-access-ndppn\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:20:35.868764 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:35.868763 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7490cfd5-895e-4d7b-a842-02bf605da03f-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:20:36.472434 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:36.472406 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6eb88-thnxg" Oct 30 00:20:36.472625 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:36.472442 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6eb88-thnxg" event={"ID":"7490cfd5-895e-4d7b-a842-02bf605da03f","Type":"ContainerDied","Data":"39d1183736941fcb07f0e4fed171907899a8ed3462b0bbdff7078dc1c9ab1146"} Oct 30 00:20:36.472625 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:36.472490 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39d1183736941fcb07f0e4fed171907899a8ed3462b0bbdff7078dc1c9ab1146" Oct 30 00:20:36.474400 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:36.474369 5001 generic.go:356] "Generic (PLEG): container finished" podID="cad7150c-93d8-4200-abbb-a50f1d7449a2" containerID="ddf487abfd86f974827726f1c8f710c14f4fb25ffd16efab8fc46f4fc756a1e3" exitCode=0 Oct 30 00:20:36.474538 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:36.474458 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-914a49ab-x96qs" event={"ID":"cad7150c-93d8-4200-abbb-a50f1d7449a2","Type":"ContainerDied","Data":"ddf487abfd86f974827726f1c8f710c14f4fb25ffd16efab8fc46f4fc756a1e3"} Oct 30 00:20:36.654567 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:36.654541 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe8ad5-qjctz" Oct 30 00:20:36.784568 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:36.784529 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e23c62e9-023c-4ff6-9c35-b724da9a824d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e23c62e9-023c-4ff6-9c35-b724da9a824d-maas-api-xks-ca\") pod \"e23c62e9-023c-4ff6-9c35-b724da9a824d\" (UID: \"e23c62e9-023c-4ff6-9c35-b724da9a824d\") " Oct 30 00:20:36.784749 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:36.784659 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e23c62e9-023c-4ff6-9c35-b724da9a824d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e23c62e9-023c-4ff6-9c35-b724da9a824d-maas-api-openshift-ca\") pod \"e23c62e9-023c-4ff6-9c35-b724da9a824d\" (UID: \"e23c62e9-023c-4ff6-9c35-b724da9a824d\") " Oct 30 00:20:36.784749 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:36.784713 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e23c62e9-023c-4ff6-9c35-b724da9a824d-kube-api-access-sghrn\" (UniqueName: \"kubernetes.io/projected/e23c62e9-023c-4ff6-9c35-b724da9a824d-kube-api-access-sghrn\") pod \"e23c62e9-023c-4ff6-9c35-b724da9a824d\" (UID: \"e23c62e9-023c-4ff6-9c35-b724da9a824d\") " Oct 30 00:20:36.785163 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:36.785127 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e23c62e9-023c-4ff6-9c35-b724da9a824d-maas-api-openshift-ca" pod "e23c62e9-023c-4ff6-9c35-b724da9a824d" (UID: "e23c62e9-023c-4ff6-9c35-b724da9a824d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:20:36.786692 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:36.786663 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e23c62e9-023c-4ff6-9c35-b724da9a824d-maas-api-xks-ca" pod "e23c62e9-023c-4ff6-9c35-b724da9a824d" (UID: "e23c62e9-023c-4ff6-9c35-b724da9a824d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:20:36.786992 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:36.786965 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e23c62e9-023c-4ff6-9c35-b724da9a824d-kube-api-access-sghrn" pod "e23c62e9-023c-4ff6-9c35-b724da9a824d" (UID: "e23c62e9-023c-4ff6-9c35-b724da9a824d"). InnerVolumeSpecName "kube-api-access-sghrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:20:36.886279 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:36.886244 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e23c62e9-023c-4ff6-9c35-b724da9a824d-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:20:36.886279 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:36.886276 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sghrn\" (UniqueName: \"kubernetes.io/projected/e23c62e9-023c-4ff6-9c35-b724da9a824d-kube-api-access-sghrn\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:20:36.886279 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:36.886286 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e23c62e9-023c-4ff6-9c35-b724da9a824d-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:20:37.489867 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:37.489779 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe8ad5-qjctz" Oct 30 00:20:37.489867 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:37.489808 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe8ad5-qjctz" event={"ID":"e23c62e9-023c-4ff6-9c35-b724da9a824d","Type":"ContainerDied","Data":"d851e17d0741761e30a6a05b093dd246c4a88841ba1a6cf7e87600ef673cca8b"} Oct 30 00:20:37.490196 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:37.489954 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d851e17d0741761e30a6a05b093dd246c4a88841ba1a6cf7e87600ef673cca8b" Oct 30 00:20:37.760056 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:37.759998 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-914a49ab-x96qs" Oct 30 00:20:37.851385 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:37.851355 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cad7150c-93d8-4200-abbb-a50f1d7449a2-kube-api-access-rth9m\" (UniqueName: \"kubernetes.io/projected/cad7150c-93d8-4200-abbb-a50f1d7449a2-kube-api-access-rth9m\") pod \"cad7150c-93d8-4200-abbb-a50f1d7449a2\" (UID: \"cad7150c-93d8-4200-abbb-a50f1d7449a2\") " Oct 30 00:20:37.851576 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:37.851459 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cad7150c-93d8-4200-abbb-a50f1d7449a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cad7150c-93d8-4200-abbb-a50f1d7449a2-maas-api-xks-ca\") pod \"cad7150c-93d8-4200-abbb-a50f1d7449a2\" (UID: \"cad7150c-93d8-4200-abbb-a50f1d7449a2\") " Oct 30 00:20:37.851576 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:37.851520 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cad7150c-93d8-4200-abbb-a50f1d7449a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cad7150c-93d8-4200-abbb-a50f1d7449a2-maas-api-openshift-ca\") pod \"cad7150c-93d8-4200-abbb-a50f1d7449a2\" (UID: \"cad7150c-93d8-4200-abbb-a50f1d7449a2\") " Oct 30 00:20:37.851979 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:37.851949 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad7150c-93d8-4200-abbb-a50f1d7449a2-maas-api-openshift-ca" pod "cad7150c-93d8-4200-abbb-a50f1d7449a2" (UID: "cad7150c-93d8-4200-abbb-a50f1d7449a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:20:37.852136 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:37.852123 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cad7150c-93d8-4200-abbb-a50f1d7449a2-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:20:37.853568 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:37.853546 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad7150c-93d8-4200-abbb-a50f1d7449a2-kube-api-access-rth9m" pod "cad7150c-93d8-4200-abbb-a50f1d7449a2" (UID: "cad7150c-93d8-4200-abbb-a50f1d7449a2"). InnerVolumeSpecName "kube-api-access-rth9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:20:37.854014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:37.853993 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cad7150c-93d8-4200-abbb-a50f1d7449a2-maas-api-xks-ca" pod "cad7150c-93d8-4200-abbb-a50f1d7449a2" (UID: "cad7150c-93d8-4200-abbb-a50f1d7449a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:20:37.957042 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:37.953686 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cad7150c-93d8-4200-abbb-a50f1d7449a2-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:20:37.957042 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:37.953720 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rth9m\" (UniqueName: \"kubernetes.io/projected/cad7150c-93d8-4200-abbb-a50f1d7449a2-kube-api-access-rth9m\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:20:38.065071 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.060559 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8"] Oct 30 00:20:38.065071 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.061531 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e23c62e9-023c-4ff6-9c35-b724da9a824d" containerName="revoke-keys" Oct 30 00:20:38.065071 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.061551 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e23c62e9-023c-4ff6-9c35-b724da9a824d" containerName="revoke-keys" Oct 30 00:20:38.065071 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.061586 5001 cpu_manager.go:414] "containerMap: removing container" podUID="7490cfd5-895e-4d7b-a842-02bf605da03f" containerName="revoke-keys" Oct 30 00:20:38.065071 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.061596 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7490cfd5-895e-4d7b-a842-02bf605da03f" containerName="revoke-keys" Oct 30 00:20:38.065071 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.061624 5001 cpu_manager.go:414] "containerMap: removing container" podUID="cad7150c-93d8-4200-abbb-a50f1d7449a2" containerName="revoke-keys" Oct 30 00:20:38.065071 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.061633 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cad7150c-93d8-4200-abbb-a50f1d7449a2" containerName="revoke-keys" Oct 30 00:20:38.065071 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.061809 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e23c62e9-023c-4ff6-9c35-b724da9a824d" containerName="revoke-keys" Oct 30 00:20:38.065071 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.061827 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7490cfd5-895e-4d7b-a842-02bf605da03f" containerName="revoke-keys" Oct 30 00:20:38.065071 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.061838 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="cad7150c-93d8-4200-abbb-a50f1d7449a2" containerName="revoke-keys" Oct 30 00:20:38.073122 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.067024 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:20:38.086437 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.086404 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8"] Oct 30 00:20:38.098971 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.098946 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-7417dc-openshift-default-dockercfg-j7dfg\"" Oct 30 00:20:38.183939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.183819 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istio-podinfo\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:20:38.183939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.183898 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dtr2\" (UniqueName: \"kubernetes.io/projected/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-kube-api-access-4dtr2\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:20:38.184200 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.183956 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istio-token\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:20:38.184200 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.184071 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istio-data\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:20:38.184200 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.184135 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-workload-certs\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:20:38.184363 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.184200 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istio-envoy\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:20:38.184363 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.184272 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-workload-socket\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:20:38.184363 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.184316 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istiod-ca-cert\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:20:38.184538 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.184400 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-credential-socket\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:20:38.307939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.302853 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-workload-certs\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:20:38.307939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.302964 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istio-envoy\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:20:38.307939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.303031 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-workload-socket\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:20:38.307939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.303070 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istiod-ca-cert\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:20:38.307939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.303148 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-credential-socket\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:20:38.307939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.303297 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istio-podinfo\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:20:38.307939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.303336 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dtr2\" (UniqueName: \"kubernetes.io/projected/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-kube-api-access-4dtr2\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:20:38.307939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.303390 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istio-token\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:20:38.307939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.303489 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istio-data\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:20:38.307939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.303968 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istio-data\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:20:38.307939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.304212 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-workload-certs\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:20:38.307939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.304846 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-credential-socket\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:20:38.307939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.305116 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-workload-socket\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:20:38.307939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.305710 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istiod-ca-cert\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:20:38.313933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.311462 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istio-envoy\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:20:38.324974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.320713 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istio-podinfo\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:20:38.359983 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.354518 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istio-token\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:20:38.376222 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.372100 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dtr2\" (UniqueName: \"kubernetes.io/projected/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-kube-api-access-4dtr2\") pod \"e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:20:38.402089 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.401858 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:20:38.601171 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.601057 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-914a49ab-x96qs" event={"ID":"cad7150c-93d8-4200-abbb-a50f1d7449a2","Type":"ContainerDied","Data":"3af359ade1f8891407ebba2a14e801b705be4a2919dba8b3c969cd567f24e8ae"} Oct 30 00:20:38.601443 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.601414 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3af359ade1f8891407ebba2a14e801b705be4a2919dba8b3c969cd567f24e8ae" Oct 30 00:20:38.605929 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:38.601732 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-914a49ab-x96qs" Oct 30 00:20:38.663816 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:20:38.663681 5001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad7150c_93d8_4200_abbb_a50f1d7449a2.slice\": RecentStats: unable to find data in memory cache]" Oct 30 00:20:39.061635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:20:39.061600 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f84a41_7ac2_4734_ab36_ebf29a62b8fa.slice/crio-70731f4e20f7530c4f879baad5bff1535877df001a81d3cbd0efe751e24350cb WatchSource:0}: Error finding container 70731f4e20f7530c4f879baad5bff1535877df001a81d3cbd0efe751e24350cb: Status 404 returned error can't find the container with id 70731f4e20f7530c4f879baad5bff1535877df001a81d3cbd0efe751e24350cb Oct 30 00:20:39.063374 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:39.063345 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8"] Oct 30 00:20:39.065080 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:39.065035 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:20:39.065209 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:39.065124 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:20:39.065209 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:39.065166 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:20:39.614138 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:39.614096 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" event={"ID":"97f84a41-7ac2-4734-ab36-ebf29a62b8fa","Type":"ContainerStarted","Data":"4068448d5f41d3e76598637ce899a36296453078dbfb02141c2fbcd30862b98d"} Oct 30 00:20:39.614343 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:39.614194 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" event={"ID":"97f84a41-7ac2-4734-ab36-ebf29a62b8fa","Type":"ContainerStarted","Data":"70731f4e20f7530c4f879baad5bff1535877df001a81d3cbd0efe751e24350cb"} Oct 30 00:20:39.656969 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:39.656902 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" podStartSLOduration=1.656888513 podStartE2EDuration="1.656888513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:20:39.653412416 +0000 UTC m=+954.858913596" watchObservedRunningTime="2026-10-30 00:20:39.656888513 +0000 UTC m=+954.862389694" Oct 30 00:20:40.407606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:40.407570 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:20:40.413163 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:40.413135 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:20:40.623568 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:40.623531 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:20:40.624360 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:40.624338 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:20:43.667876 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:43.667842 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7417dc-cdd68c66c-cjxww"] Oct 30 00:20:43.674145 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:43.674106 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7417dc-cdd68c66c-cjxww" Oct 30 00:20:43.691606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:43.691571 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7417dc-cdd68c66c-cjxww"] Oct 30 00:20:43.695258 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:43.695232 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-7417dc\"" Oct 30 00:20:43.695873 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:43.695851 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-7417dc-dockercfg-67vkp\"" Oct 30 00:20:43.716794 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:43.716756 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j87bt\" (UniqueName: \"kubernetes.io/projected/c115ecb3-2006-4756-a8cf-d06a56ebd4f7-kube-api-access-j87bt\") pod \"payload-pre-processing-e2e-api-b-7417dc-cdd68c66c-cjxww\" (UID: \"c115ecb3-2006-4756-a8cf-d06a56ebd4f7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7417dc-cdd68c66c-cjxww" Oct 30 00:20:43.716978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:43.716827 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c115ecb3-2006-4756-a8cf-d06a56ebd4f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7417dc-cdd68c66c-cjxww\" (UID: \"c115ecb3-2006-4756-a8cf-d06a56ebd4f7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7417dc-cdd68c66c-cjxww" Oct 30 00:20:43.854645 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:43.833874 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j87bt\" (UniqueName: \"kubernetes.io/projected/c115ecb3-2006-4756-a8cf-d06a56ebd4f7-kube-api-access-j87bt\") pod \"payload-pre-processing-e2e-api-b-7417dc-cdd68c66c-cjxww\" (UID: \"c115ecb3-2006-4756-a8cf-d06a56ebd4f7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7417dc-cdd68c66c-cjxww" Oct 30 00:20:43.854980 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:43.854951 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c115ecb3-2006-4756-a8cf-d06a56ebd4f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7417dc-cdd68c66c-cjxww\" (UID: \"c115ecb3-2006-4756-a8cf-d06a56ebd4f7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7417dc-cdd68c66c-cjxww" Oct 30 00:20:43.887936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:43.855985 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c115ecb3-2006-4756-a8cf-d06a56ebd4f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7417dc-cdd68c66c-cjxww\" (UID: \"c115ecb3-2006-4756-a8cf-d06a56ebd4f7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7417dc-cdd68c66c-cjxww" Oct 30 00:20:43.995999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:43.985705 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j87bt\" (UniqueName: \"kubernetes.io/projected/c115ecb3-2006-4756-a8cf-d06a56ebd4f7-kube-api-access-j87bt\") pod \"payload-pre-processing-e2e-api-b-7417dc-cdd68c66c-cjxww\" (UID: \"c115ecb3-2006-4756-a8cf-d06a56ebd4f7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7417dc-cdd68c66c-cjxww" Oct 30 00:20:43.995999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:43.994140 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7417dc-cdd68c66c-cjxww" Oct 30 00:20:44.454192 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:44.454160 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7417dc-cdd68c66c-cjxww"] Oct 30 00:20:44.454849 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:20:44.454819 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc115ecb3_2006_4756_a8cf_d06a56ebd4f7.slice/crio-8041d50accf15e086cef3cd2a02d1f26807ea6a4741602b85c9e3b532ce093e0 WatchSource:0}: Error finding container 8041d50accf15e086cef3cd2a02d1f26807ea6a4741602b85c9e3b532ce093e0: Status 404 returned error can't find the container with id 8041d50accf15e086cef3cd2a02d1f26807ea6a4741602b85c9e3b532ce093e0 Oct 30 00:20:44.659627 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:44.659589 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7417dc-cdd68c66c-cjxww" event={"ID":"c115ecb3-2006-4756-a8cf-d06a56ebd4f7","Type":"ContainerStarted","Data":"9630c2c6b484fac1bb9cf840dd489f9b75e039dc86adf4c61230a17606287bd1"} Oct 30 00:20:44.659837 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:44.659656 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7417dc-cdd68c66c-cjxww" Oct 30 00:20:44.659837 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:44.659680 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7417dc-cdd68c66c-cjxww" event={"ID":"c115ecb3-2006-4756-a8cf-d06a56ebd4f7","Type":"ContainerStarted","Data":"8041d50accf15e086cef3cd2a02d1f26807ea6a4741602b85c9e3b532ce093e0"} Oct 30 00:20:44.702390 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:44.702292 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7417dc-cdd68c66c-cjxww" podStartSLOduration=1.702275438 podStartE2EDuration="1.702275438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:20:44.697177792 +0000 UTC m=+959.902678972" watchObservedRunningTime="2026-10-30 00:20:44.702275438 +0000 UTC m=+959.907776619" Oct 30 00:20:45.755902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:45.755862 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp"] Oct 30 00:20:45.762022 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:45.761999 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:20:45.799320 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:45.799284 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp"] Oct 30 00:20:45.810043 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:45.810010 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-461609-openshift-default-dockercfg-pdfzs\"" Oct 30 00:20:45.954983 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:45.953738 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-workload-socket\") pod \"e2e-api-a-461609-openshift-default-86bb859445-gnfjp\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:20:45.954983 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:45.953791 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-istio-data\") pod \"e2e-api-a-461609-openshift-default-86bb859445-gnfjp\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:20:45.954983 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:45.953823 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1584ba56-ffef-49e1-8edd-399ca7b7f366-istio-podinfo\") pod \"e2e-api-a-461609-openshift-default-86bb859445-gnfjp\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:20:45.954983 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:45.953943 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cxxq\" (UniqueName: \"kubernetes.io/projected/1584ba56-ffef-49e1-8edd-399ca7b7f366-kube-api-access-7cxxq\") pod \"e2e-api-a-461609-openshift-default-86bb859445-gnfjp\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:20:45.954983 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:45.954076 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1584ba56-ffef-49e1-8edd-399ca7b7f366-istiod-ca-cert\") pod \"e2e-api-a-461609-openshift-default-86bb859445-gnfjp\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:20:45.954983 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:45.954102 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-istio-envoy\") pod \"e2e-api-a-461609-openshift-default-86bb859445-gnfjp\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:20:45.954983 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:45.954156 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-credential-socket\") pod \"e2e-api-a-461609-openshift-default-86bb859445-gnfjp\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:20:45.954983 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:45.954184 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1584ba56-ffef-49e1-8edd-399ca7b7f366-istio-token\") pod \"e2e-api-a-461609-openshift-default-86bb859445-gnfjp\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:20:45.954983 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:45.954231 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-workload-certs\") pod \"e2e-api-a-461609-openshift-default-86bb859445-gnfjp\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:20:46.058934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:46.057866 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1584ba56-ffef-49e1-8edd-399ca7b7f366-istiod-ca-cert\") pod \"e2e-api-a-461609-openshift-default-86bb859445-gnfjp\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:20:46.058934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:46.057933 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-istio-envoy\") pod \"e2e-api-a-461609-openshift-default-86bb859445-gnfjp\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:20:46.058934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:46.058005 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-credential-socket\") pod \"e2e-api-a-461609-openshift-default-86bb859445-gnfjp\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:20:46.058934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:46.058034 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1584ba56-ffef-49e1-8edd-399ca7b7f366-istio-token\") pod \"e2e-api-a-461609-openshift-default-86bb859445-gnfjp\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:20:46.058934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:46.058080 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-workload-certs\") pod \"e2e-api-a-461609-openshift-default-86bb859445-gnfjp\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:20:46.058934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:46.058161 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-workload-socket\") pod \"e2e-api-a-461609-openshift-default-86bb859445-gnfjp\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:20:46.058934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:46.058197 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-istio-data\") pod \"e2e-api-a-461609-openshift-default-86bb859445-gnfjp\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:20:46.058934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:46.058220 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1584ba56-ffef-49e1-8edd-399ca7b7f366-istio-podinfo\") pod \"e2e-api-a-461609-openshift-default-86bb859445-gnfjp\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:20:46.058934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:46.058371 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cxxq\" (UniqueName: \"kubernetes.io/projected/1584ba56-ffef-49e1-8edd-399ca7b7f366-kube-api-access-7cxxq\") pod \"e2e-api-a-461609-openshift-default-86bb859445-gnfjp\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:20:46.062935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:46.060156 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1584ba56-ffef-49e1-8edd-399ca7b7f366-istiod-ca-cert\") pod \"e2e-api-a-461609-openshift-default-86bb859445-gnfjp\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:20:46.062935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:46.060780 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-credential-socket\") pod \"e2e-api-a-461609-openshift-default-86bb859445-gnfjp\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:20:46.062935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:46.061648 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-istio-data\") pod \"e2e-api-a-461609-openshift-default-86bb859445-gnfjp\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:20:46.062935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:46.062338 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-workload-certs\") pod \"e2e-api-a-461609-openshift-default-86bb859445-gnfjp\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:20:46.067407 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:46.065004 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-workload-socket\") pod \"e2e-api-a-461609-openshift-default-86bb859445-gnfjp\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:20:46.067407 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:46.065483 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1584ba56-ffef-49e1-8edd-399ca7b7f366-istio-podinfo\") pod \"e2e-api-a-461609-openshift-default-86bb859445-gnfjp\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:20:46.088032 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:46.087736 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cxxq\" (UniqueName: \"kubernetes.io/projected/1584ba56-ffef-49e1-8edd-399ca7b7f366-kube-api-access-7cxxq\") pod \"e2e-api-a-461609-openshift-default-86bb859445-gnfjp\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:20:46.090609 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:46.090551 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1584ba56-ffef-49e1-8edd-399ca7b7f366-istio-token\") pod \"e2e-api-a-461609-openshift-default-86bb859445-gnfjp\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:20:46.091350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:46.091327 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-istio-envoy\") pod \"e2e-api-a-461609-openshift-default-86bb859445-gnfjp\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:20:46.382922 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:46.382816 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:20:47.368133 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:47.362570 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp"] Oct 30 00:20:47.415740 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:47.415679 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:20:47.416049 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:47.416025 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:20:47.416230 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:47.416204 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:20:47.571681 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:20:47.570151 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:20:48.011450 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:48.011415 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" event={"ID":"1584ba56-ffef-49e1-8edd-399ca7b7f366","Type":"ContainerStarted","Data":"68562c9c74560a1ba68b6a3474a65cfdebcc91dd3793ba80d25924a386d06e26"} Oct 30 00:20:48.011612 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:48.011472 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" event={"ID":"1584ba56-ffef-49e1-8edd-399ca7b7f366","Type":"ContainerStarted","Data":"c50c4d02603cc2266adeb954918be7ac8c47c393aff415d48b4deb21339f414a"} Oct 30 00:20:48.076904 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:48.076837 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" podStartSLOduration=3.076818481 podStartE2EDuration="3.076818481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:20:48.060484558 +0000 UTC m=+963.265985738" watchObservedRunningTime="2026-10-30 00:20:48.076818481 +0000 UTC m=+963.282319667" Oct 30 00:20:48.177346 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:48.177311 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5368f66-fltv9"] Oct 30 00:20:48.182964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:48.182565 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5368f66-fltv9" Oct 30 00:20:48.204935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:48.201571 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:20:48.204935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:48.201782 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:20:48.204935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:48.201849 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:20:48.204935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:48.201960 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:20:48.228946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:48.218437 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5368f66-fltv9"] Oct 30 00:20:48.363617 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:48.363499 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/628e474e-57df-4912-a13c-9c83ab3b0086-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5368f66-fltv9\" (UID: \"628e474e-57df-4912-a13c-9c83ab3b0086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5368f66-fltv9" Oct 30 00:20:48.363791 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:48.363642 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/628e474e-57df-4912-a13c-9c83ab3b0086-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5368f66-fltv9\" (UID: \"628e474e-57df-4912-a13c-9c83ab3b0086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5368f66-fltv9" Oct 30 00:20:48.363791 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:48.363694 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gsfv\" (UniqueName: \"kubernetes.io/projected/628e474e-57df-4912-a13c-9c83ab3b0086-kube-api-access-5gsfv\") pod \"maas-api-revoke-sub-d5368f66-fltv9\" (UID: \"628e474e-57df-4912-a13c-9c83ab3b0086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5368f66-fltv9" Oct 30 00:20:48.388946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:48.387824 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:20:48.398254 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:48.392554 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:20:48.464896 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:48.464860 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/628e474e-57df-4912-a13c-9c83ab3b0086-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5368f66-fltv9\" (UID: \"628e474e-57df-4912-a13c-9c83ab3b0086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5368f66-fltv9" Oct 30 00:20:48.465129 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:48.465017 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/628e474e-57df-4912-a13c-9c83ab3b0086-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5368f66-fltv9\" (UID: \"628e474e-57df-4912-a13c-9c83ab3b0086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5368f66-fltv9" Oct 30 00:20:48.465129 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:48.465068 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gsfv\" (UniqueName: \"kubernetes.io/projected/628e474e-57df-4912-a13c-9c83ab3b0086-kube-api-access-5gsfv\") pod \"maas-api-revoke-sub-d5368f66-fltv9\" (UID: \"628e474e-57df-4912-a13c-9c83ab3b0086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5368f66-fltv9" Oct 30 00:20:48.465709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:48.465689 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/628e474e-57df-4912-a13c-9c83ab3b0086-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5368f66-fltv9\" (UID: \"628e474e-57df-4912-a13c-9c83ab3b0086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5368f66-fltv9" Oct 30 00:20:48.467329 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:48.467305 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/628e474e-57df-4912-a13c-9c83ab3b0086-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5368f66-fltv9\" (UID: \"628e474e-57df-4912-a13c-9c83ab3b0086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5368f66-fltv9" Oct 30 00:20:48.486117 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:48.486080 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gsfv\" (UniqueName: \"kubernetes.io/projected/628e474e-57df-4912-a13c-9c83ab3b0086-kube-api-access-5gsfv\") pod \"maas-api-revoke-sub-d5368f66-fltv9\" (UID: \"628e474e-57df-4912-a13c-9c83ab3b0086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5368f66-fltv9" Oct 30 00:20:48.540442 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:48.540413 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5368f66-fltv9" Oct 30 00:20:48.938663 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:20:48.938631 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod628e474e_57df_4912_a13c_9c83ab3b0086.slice/crio-b19ee8b457e2ba659671e72ba3df8eb6c4838cd106c0007afde1896d6e9bd4be WatchSource:0}: Error finding container b19ee8b457e2ba659671e72ba3df8eb6c4838cd106c0007afde1896d6e9bd4be: Status 404 returned error can't find the container with id b19ee8b457e2ba659671e72ba3df8eb6c4838cd106c0007afde1896d6e9bd4be Oct 30 00:20:48.939064 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:48.939043 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5368f66-fltv9"] Oct 30 00:20:49.026246 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:49.026207 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5368f66-fltv9" event={"ID":"628e474e-57df-4912-a13c-9c83ab3b0086","Type":"ContainerStarted","Data":"b19ee8b457e2ba659671e72ba3df8eb6c4838cd106c0007afde1896d6e9bd4be"} Oct 30 00:20:49.026523 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:49.026504 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:20:49.027603 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:49.027578 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:20:50.036756 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:50.036714 5001 generic.go:356] "Generic (PLEG): container finished" podID="628e474e-57df-4912-a13c-9c83ab3b0086" containerID="0f0efa84858f2ffee9c9b9e2c078265ba81920ca508a5984030947c68ce7b87e" exitCode=0 Oct 30 00:20:50.037245 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:50.036798 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5368f66-fltv9" event={"ID":"628e474e-57df-4912-a13c-9c83ab3b0086","Type":"ContainerDied","Data":"0f0efa84858f2ffee9c9b9e2c078265ba81920ca508a5984030947c68ce7b87e"} Oct 30 00:20:51.416103 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:51.416079 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5368f66-fltv9" Oct 30 00:20:51.435778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:51.435750 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/628e474e-57df-4912-a13c-9c83ab3b0086-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/628e474e-57df-4912-a13c-9c83ab3b0086-maas-api-openshift-ca\") pod \"628e474e-57df-4912-a13c-9c83ab3b0086\" (UID: \"628e474e-57df-4912-a13c-9c83ab3b0086\") " Oct 30 00:20:51.435956 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:51.435941 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/628e474e-57df-4912-a13c-9c83ab3b0086-kube-api-access-5gsfv\" (UniqueName: \"kubernetes.io/projected/628e474e-57df-4912-a13c-9c83ab3b0086-kube-api-access-5gsfv\") pod \"628e474e-57df-4912-a13c-9c83ab3b0086\" (UID: \"628e474e-57df-4912-a13c-9c83ab3b0086\") " Oct 30 00:20:51.436036 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:51.436022 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/628e474e-57df-4912-a13c-9c83ab3b0086-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/628e474e-57df-4912-a13c-9c83ab3b0086-maas-api-xks-ca\") pod \"628e474e-57df-4912-a13c-9c83ab3b0086\" (UID: \"628e474e-57df-4912-a13c-9c83ab3b0086\") " Oct 30 00:20:51.436181 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:51.436154 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/628e474e-57df-4912-a13c-9c83ab3b0086-maas-api-openshift-ca" pod "628e474e-57df-4912-a13c-9c83ab3b0086" (UID: "628e474e-57df-4912-a13c-9c83ab3b0086"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:20:51.436596 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:51.436578 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/628e474e-57df-4912-a13c-9c83ab3b0086-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:20:51.438063 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:51.438040 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/628e474e-57df-4912-a13c-9c83ab3b0086-maas-api-xks-ca" pod "628e474e-57df-4912-a13c-9c83ab3b0086" (UID: "628e474e-57df-4912-a13c-9c83ab3b0086"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:20:51.438169 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:51.438074 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/628e474e-57df-4912-a13c-9c83ab3b0086-kube-api-access-5gsfv" pod "628e474e-57df-4912-a13c-9c83ab3b0086" (UID: "628e474e-57df-4912-a13c-9c83ab3b0086"). InnerVolumeSpecName "kube-api-access-5gsfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:20:51.537545 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:51.537514 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gsfv\" (UniqueName: \"kubernetes.io/projected/628e474e-57df-4912-a13c-9c83ab3b0086-kube-api-access-5gsfv\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:20:51.537545 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:51.537544 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/628e474e-57df-4912-a13c-9c83ab3b0086-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:20:52.104393 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.104366 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5368f66-fltv9" Oct 30 00:20:52.104576 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.104405 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5368f66-fltv9" event={"ID":"628e474e-57df-4912-a13c-9c83ab3b0086","Type":"ContainerDied","Data":"b19ee8b457e2ba659671e72ba3df8eb6c4838cd106c0007afde1896d6e9bd4be"} Oct 30 00:20:52.104576 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.104454 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b19ee8b457e2ba659671e72ba3df8eb6c4838cd106c0007afde1896d6e9bd4be" Oct 30 00:20:52.166773 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.166746 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e2a011-dd5bd7995-m4h7w"] Oct 30 00:20:52.167424 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.167408 5001 cpu_manager.go:414] "containerMap: removing container" podUID="628e474e-57df-4912-a13c-9c83ab3b0086" containerName="revoke-keys" Oct 30 00:20:52.167491 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.167425 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="628e474e-57df-4912-a13c-9c83ab3b0086" containerName="revoke-keys" Oct 30 00:20:52.167564 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.167555 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="628e474e-57df-4912-a13c-9c83ab3b0086" containerName="revoke-keys" Oct 30 00:20:52.171061 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.171046 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e2a011-dd5bd7995-m4h7w" Oct 30 00:20:52.186844 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.186819 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e2a011-dockercfg-sm48w\"" Oct 30 00:20:52.189808 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.189786 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e2a011-dd5bd7995-m4h7w"] Oct 30 00:20:52.193349 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.193330 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e2a011\"" Oct 30 00:20:52.247754 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.247717 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3b2f8d-f573-4956-8b85-504d533c8fce-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e2a011-dd5bd7995-m4h7w\" (UID: \"1b3b2f8d-f573-4956-8b85-504d533c8fce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e2a011-dd5bd7995-m4h7w" Oct 30 00:20:52.247921 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.247772 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll44h\" (UniqueName: \"kubernetes.io/projected/1b3b2f8d-f573-4956-8b85-504d533c8fce-kube-api-access-ll44h\") pod \"payload-pre-processing-e2e-api-a-e2a011-dd5bd7995-m4h7w\" (UID: \"1b3b2f8d-f573-4956-8b85-504d533c8fce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e2a011-dd5bd7995-m4h7w" Oct 30 00:20:52.397984 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.397351 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3b2f8d-f573-4956-8b85-504d533c8fce-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e2a011-dd5bd7995-m4h7w\" (UID: \"1b3b2f8d-f573-4956-8b85-504d533c8fce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e2a011-dd5bd7995-m4h7w" Oct 30 00:20:52.397984 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.397439 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll44h\" (UniqueName: \"kubernetes.io/projected/1b3b2f8d-f573-4956-8b85-504d533c8fce-kube-api-access-ll44h\") pod \"payload-pre-processing-e2e-api-a-e2a011-dd5bd7995-m4h7w\" (UID: \"1b3b2f8d-f573-4956-8b85-504d533c8fce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e2a011-dd5bd7995-m4h7w" Oct 30 00:20:52.405771 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.404311 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3b2f8d-f573-4956-8b85-504d533c8fce-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e2a011-dd5bd7995-m4h7w\" (UID: \"1b3b2f8d-f573-4956-8b85-504d533c8fce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e2a011-dd5bd7995-m4h7w" Oct 30 00:20:52.491068 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.490980 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll44h\" (UniqueName: \"kubernetes.io/projected/1b3b2f8d-f573-4956-8b85-504d533c8fce-kube-api-access-ll44h\") pod \"payload-pre-processing-e2e-api-a-e2a011-dd5bd7995-m4h7w\" (UID: \"1b3b2f8d-f573-4956-8b85-504d533c8fce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e2a011-dd5bd7995-m4h7w" Oct 30 00:20:52.786602 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.786565 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e2a011-dd5bd7995-m4h7w" Oct 30 00:20:52.806745 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.806706 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds"] Oct 30 00:20:52.813715 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.813682 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" Oct 30 00:20:52.817610 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.817581 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3e66631-9562-4bb5-b005-1fc179a8582d-tmp-dir\") pod \"e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds\" (UID: \"b3e66631-9562-4bb5-b005-1fc179a8582d\") " pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" Oct 30 00:20:52.817834 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.817641 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3e66631-9562-4bb5-b005-1fc179a8582d-home\") pod \"e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds\" (UID: \"b3e66631-9562-4bb5-b005-1fc179a8582d\") " pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" Oct 30 00:20:52.817834 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.817741 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvbdl\" (UniqueName: \"kubernetes.io/projected/b3e66631-9562-4bb5-b005-1fc179a8582d-kube-api-access-mvbdl\") pod \"e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds\" (UID: \"b3e66631-9562-4bb5-b005-1fc179a8582d\") " pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" Oct 30 00:20:52.817834 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.817797 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3e66631-9562-4bb5-b005-1fc179a8582d-dshm\") pod \"e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds\" (UID: \"b3e66631-9562-4bb5-b005-1fc179a8582d\") " pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" Oct 30 00:20:52.818106 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.817950 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3e66631-9562-4bb5-b005-1fc179a8582d-model-cache\") pod \"e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds\" (UID: \"b3e66631-9562-4bb5-b005-1fc179a8582d\") " pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" Oct 30 00:20:52.818106 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.818075 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3e66631-9562-4bb5-b005-1fc179a8582d-tls-certs\") pod \"e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds\" (UID: \"b3e66631-9562-4bb5-b005-1fc179a8582d\") " pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" Oct 30 00:20:52.829143 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.829111 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-0847e8c0\"/\"openshift-service-ca.crt\"" Oct 30 00:20:52.836403 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.835026 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-0847e8c0\"/\"e2e-lifecycle-model-0847e8c0-kserve-self-signed-certs\"" Oct 30 00:20:52.836403 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.835170 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-0847e8c0\"/\"default-dockercfg-trrsq\"" Oct 30 00:20:52.836403 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.835242 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-0847e8c0\"/\"kube-root-ca.crt\"" Oct 30 00:20:52.867178 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.867045 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds"] Oct 30 00:20:52.924437 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.924393 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3e66631-9562-4bb5-b005-1fc179a8582d-tmp-dir\") pod \"e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds\" (UID: \"b3e66631-9562-4bb5-b005-1fc179a8582d\") " pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" Oct 30 00:20:52.924602 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.924462 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3e66631-9562-4bb5-b005-1fc179a8582d-home\") pod \"e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds\" (UID: \"b3e66631-9562-4bb5-b005-1fc179a8582d\") " pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" Oct 30 00:20:52.924602 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.924563 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvbdl\" (UniqueName: \"kubernetes.io/projected/b3e66631-9562-4bb5-b005-1fc179a8582d-kube-api-access-mvbdl\") pod \"e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds\" (UID: \"b3e66631-9562-4bb5-b005-1fc179a8582d\") " pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" Oct 30 00:20:52.924725 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.924615 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3e66631-9562-4bb5-b005-1fc179a8582d-dshm\") pod \"e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds\" (UID: \"b3e66631-9562-4bb5-b005-1fc179a8582d\") " pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" Oct 30 00:20:52.924778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.924749 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3e66631-9562-4bb5-b005-1fc179a8582d-model-cache\") pod \"e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds\" (UID: \"b3e66631-9562-4bb5-b005-1fc179a8582d\") " pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" Oct 30 00:20:52.924889 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.924866 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3e66631-9562-4bb5-b005-1fc179a8582d-tls-certs\") pod \"e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds\" (UID: \"b3e66631-9562-4bb5-b005-1fc179a8582d\") " pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" Oct 30 00:20:52.925821 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.925793 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3e66631-9562-4bb5-b005-1fc179a8582d-home\") pod \"e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds\" (UID: \"b3e66631-9562-4bb5-b005-1fc179a8582d\") " pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" Oct 30 00:20:52.926738 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.926710 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3e66631-9562-4bb5-b005-1fc179a8582d-tmp-dir\") pod \"e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds\" (UID: \"b3e66631-9562-4bb5-b005-1fc179a8582d\") " pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" Oct 30 00:20:52.927137 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.927108 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3e66631-9562-4bb5-b005-1fc179a8582d-model-cache\") pod \"e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds\" (UID: \"b3e66631-9562-4bb5-b005-1fc179a8582d\") " pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" Oct 30 00:20:52.928628 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.928606 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3e66631-9562-4bb5-b005-1fc179a8582d-tls-certs\") pod \"e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds\" (UID: \"b3e66631-9562-4bb5-b005-1fc179a8582d\") " pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" Oct 30 00:20:52.934553 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.934527 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3e66631-9562-4bb5-b005-1fc179a8582d-dshm\") pod \"e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds\" (UID: \"b3e66631-9562-4bb5-b005-1fc179a8582d\") " pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" Oct 30 00:20:52.950929 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:52.949499 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvbdl\" (UniqueName: \"kubernetes.io/projected/b3e66631-9562-4bb5-b005-1fc179a8582d-kube-api-access-mvbdl\") pod \"e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds\" (UID: \"b3e66631-9562-4bb5-b005-1fc179a8582d\") " pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" Oct 30 00:20:53.178950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:53.167443 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" Oct 30 00:20:53.251016 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:53.247100 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e2a011-dd5bd7995-m4h7w"] Oct 30 00:20:53.725303 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:53.725265 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds"] Oct 30 00:20:53.744463 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:20:53.744411 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3e66631_9562_4bb5_b005_1fc179a8582d.slice/crio-cdd4684c7c72f364c6a5b42bfc1fb246799f7be8829c919e35400234cf471016 WatchSource:0}: Error finding container cdd4684c7c72f364c6a5b42bfc1fb246799f7be8829c919e35400234cf471016: Status 404 returned error can't find the container with id cdd4684c7c72f364c6a5b42bfc1fb246799f7be8829c919e35400234cf471016 Oct 30 00:20:54.268808 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:54.268646 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e2a011-dd5bd7995-m4h7w" event={"ID":"1b3b2f8d-f573-4956-8b85-504d533c8fce","Type":"ContainerStarted","Data":"2cde9ae2c0804eb343ffc896baae6a7d08e6fd4efc49eae0e70a11a7305ac9dc"} Oct 30 00:20:54.268808 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:54.268716 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e2a011-dd5bd7995-m4h7w" event={"ID":"1b3b2f8d-f573-4956-8b85-504d533c8fce","Type":"ContainerStarted","Data":"38197857ab9e581400b1ac45dceb3ce0be2233b989cefe1ac27cde21c12a0997"} Oct 30 00:20:54.268808 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:54.268770 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e2a011-dd5bd7995-m4h7w" Oct 30 00:20:54.300019 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:54.299968 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" event={"ID":"b3e66631-9562-4bb5-b005-1fc179a8582d","Type":"ContainerStarted","Data":"cdd4684c7c72f364c6a5b42bfc1fb246799f7be8829c919e35400234cf471016"} Oct 30 00:20:54.349940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:54.342400 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e2a011-dd5bd7995-m4h7w" podStartSLOduration=2.342377281 podStartE2EDuration="2.342377281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:20:54.324170394 +0000 UTC m=+969.529671576" watchObservedRunningTime="2026-10-30 00:20:54.342377281 +0000 UTC m=+969.547878463" Oct 30 00:20:54.967961 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:54.967240 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:55.328931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:55.322497 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:55.638115 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:55.638010 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:55.706853 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:55.706814 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7417dc-cdd68c66c-cjxww" Oct 30 00:20:56.158936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:56.056164 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:56.158936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:56.059428 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:20:56.158936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:56.150043 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:56.256994 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:56.218671 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:20:56.256994 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:56.229737 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:56.256994 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:56.235100 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 30 00:20:56.428029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:56.415854 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:56.429115 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:56.429082 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:20:56.693992 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:56.668619 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:20:56.716962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:56.713397 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:20:57.057837 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:57.057056 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:57.157929 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:57.157105 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:57.157929 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:57.157839 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:57.269870 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:57.266603 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 30 00:20:57.269870 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:57.268343 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 00:20:57.269870 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:57.268628 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:20:57.380986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:57.377465 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:20:57.380986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:57.377558 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:20:57.380986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:57.377600 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:20:57.429472 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:57.427126 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:20:57.429472 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:57.427238 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:20:57.429472 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:57.427262 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cc8f4cb-hkwxz"] Oct 30 00:20:57.429472 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:57.427290 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cc8f4cb-hkwxz"] Oct 30 00:20:57.432413 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:57.432376 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 00:20:57.474623 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:57.474589 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:20:57.485838 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:57.485780 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" podUID="5e44a9bb-130f-4749-bd51-ab0fa7958247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:57.719514 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:57.718086 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:58.189937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:58.179345 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:20:58.292946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:58.292487 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:58.305430 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:58.295174 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:20:58.305430 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:58.295474 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 00:20:58.332953 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:58.318615 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:58.458990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:58.445410 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:58.472068 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:58.472019 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:58.569074 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:58.569023 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:58.721107 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:58.721006 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:20:59.053939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:59.038791 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Oct 30 00:20:59.180097 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:59.180053 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:59.235968 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:59.220824 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9de9913-86fb-4748-b043-8c0323f4e1c6" path="/var/lib/kubelet/pods/b9de9913-86fb-4748-b043-8c0323f4e1c6/volumes" Oct 30 00:20:59.236853 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:59.236823 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88f86bc7-z48bk"] Oct 30 00:20:59.301956 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:59.301425 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:59.301956 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:59.301582 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 30 00:20:59.327082 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:59.302720 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:20:59.327082 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:59.305278 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88f86bc7-z48bk"] Oct 30 00:20:59.327082 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:59.305392 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f86bc7-z48bk" Oct 30 00:20:59.327082 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:59.315787 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:59.392345 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:59.327655 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:20:59.392345 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:59.327807 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:20:59.392345 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:59.327939 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:20:59.392345 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:59.349217 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:20:59.392345 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:59.364078 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:20:59.392345 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:59.364491 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:20:59.410134 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:59.410102 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:20:59.410420 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:59.410392 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" podUID="cf513d72-64d7-4444-ae0c-fda3aef69626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:59.446933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:59.446274 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:59.474061 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:59.474017 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:20:59.496980 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:59.494574 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:20:59.653118 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:59.653024 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bec19ed-2ef0-41d3-a53c-6275eea40fb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88f86bc7-z48bk\" (UID: \"2bec19ed-2ef0-41d3-a53c-6275eea40fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f86bc7-z48bk" Oct 30 00:20:59.665409 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:59.665373 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bec19ed-2ef0-41d3-a53c-6275eea40fb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88f86bc7-z48bk\" (UID: \"2bec19ed-2ef0-41d3-a53c-6275eea40fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f86bc7-z48bk" Oct 30 00:20:59.665663 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:59.665639 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n89bj\" (UniqueName: \"kubernetes.io/projected/2bec19ed-2ef0-41d3-a53c-6275eea40fb9-kube-api-access-n89bj\") pod \"maas-api-revoke-sub-88f86bc7-z48bk\" (UID: \"2bec19ed-2ef0-41d3-a53c-6275eea40fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f86bc7-z48bk" Oct 30 00:20:59.749946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:59.738840 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:20:59.846961 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:59.846131 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bec19ed-2ef0-41d3-a53c-6275eea40fb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88f86bc7-z48bk\" (UID: \"2bec19ed-2ef0-41d3-a53c-6275eea40fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f86bc7-z48bk" Oct 30 00:20:59.846961 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:59.846183 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n89bj\" (UniqueName: \"kubernetes.io/projected/2bec19ed-2ef0-41d3-a53c-6275eea40fb9-kube-api-access-n89bj\") pod \"maas-api-revoke-sub-88f86bc7-z48bk\" (UID: \"2bec19ed-2ef0-41d3-a53c-6275eea40fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f86bc7-z48bk" Oct 30 00:20:59.846961 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:59.846334 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bec19ed-2ef0-41d3-a53c-6275eea40fb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88f86bc7-z48bk\" (UID: \"2bec19ed-2ef0-41d3-a53c-6275eea40fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f86bc7-z48bk" Oct 30 00:20:59.847311 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:59.847285 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bec19ed-2ef0-41d3-a53c-6275eea40fb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88f86bc7-z48bk\" (UID: \"2bec19ed-2ef0-41d3-a53c-6275eea40fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f86bc7-z48bk" Oct 30 00:20:59.878055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:59.878016 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:20:59.878325 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:20:59.878291 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" podUID="38fbc17b-0fb6-46c8-83af-53b83fa2bcb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:00.334939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:00.243339 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:00.393169 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:00.393119 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:00.393876 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:00.393849 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:00.441849 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:00.441808 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:21:00.482972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:00.458847 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:00.482972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:00.476012 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 00:21:00.502334 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:00.491137 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:00.559965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:00.554035 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:21:00.646979 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:00.642193 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n89bj\" (UniqueName: \"kubernetes.io/projected/2bec19ed-2ef0-41d3-a53c-6275eea40fb9-kube-api-access-n89bj\") pod \"maas-api-revoke-sub-88f86bc7-z48bk\" (UID: \"2bec19ed-2ef0-41d3-a53c-6275eea40fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f86bc7-z48bk" Oct 30 00:21:00.653556 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:00.653512 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bec19ed-2ef0-41d3-a53c-6275eea40fb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88f86bc7-z48bk\" (UID: \"2bec19ed-2ef0-41d3-a53c-6275eea40fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f86bc7-z48bk" Oct 30 00:21:00.829093 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:00.829042 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:01.025080 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:01.011343 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f86bc7-z48bk" Oct 30 00:21:01.401857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:01.400793 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 00:21:01.406570 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:01.405946 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 30 00:21:01.480353 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:01.468983 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:01.480353 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:01.478829 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:01.480353 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:01.479531 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:01.496679 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:01.496056 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:21:01.727193 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:01.710051 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:21:01.847937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:01.841259 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:21:01.864542 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:01.864072 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:01.898350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:01.896036 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:02.137933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:02.113031 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:02.240201 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:02.240103 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:21:02.266071 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:02.264338 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:21:02.266071 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:02.265757 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:21:02.285324 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:02.284411 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:21:02.299933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:02.297342 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:21:02.354388 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:02.354352 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" event={"ID":"b3e66631-9562-4bb5-b005-1fc179a8582d","Type":"ContainerStarted","Data":"ec17342524a50c2c5323dda527cdfddeb0d6c6b672677176bf4892faa6da465d"} Oct 30 00:21:02.383036 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:02.383013 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88f86bc7-z48bk"] Oct 30 00:21:02.495553 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:02.495483 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bef0d341-znp5p"] Oct 30 00:21:02.502557 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:02.502532 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef0d341-znp5p" Oct 30 00:21:02.506574 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:02.506546 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmjc8\" (UniqueName: \"kubernetes.io/projected/5eadd5f6-4577-4b82-8903-24648a5493e6-kube-api-access-zmjc8\") pod \"maas-api-revoke-sub-bef0d341-znp5p\" (UID: \"5eadd5f6-4577-4b82-8903-24648a5493e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef0d341-znp5p" Oct 30 00:21:02.506712 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:02.506689 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eadd5f6-4577-4b82-8903-24648a5493e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bef0d341-znp5p\" (UID: \"5eadd5f6-4577-4b82-8903-24648a5493e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef0d341-znp5p" Oct 30 00:21:02.506896 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:02.506878 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eadd5f6-4577-4b82-8903-24648a5493e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bef0d341-znp5p\" (UID: \"5eadd5f6-4577-4b82-8903-24648a5493e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef0d341-znp5p" Oct 30 00:21:02.543432 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:02.543370 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" podStartSLOduration=10.54335206 podStartE2EDuration="10.54335206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:02.538131571 +0000 UTC m=+977.743632754" watchObservedRunningTime="2026-10-30 00:21:02.54335206 +0000 UTC m=+977.748853241" Oct 30 00:21:02.553964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:02.553925 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bef0d341-znp5p"] Oct 30 00:21:02.631072 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:02.631030 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eadd5f6-4577-4b82-8903-24648a5493e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bef0d341-znp5p\" (UID: \"5eadd5f6-4577-4b82-8903-24648a5493e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef0d341-znp5p" Oct 30 00:21:02.632228 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:02.632196 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmjc8\" (UniqueName: \"kubernetes.io/projected/5eadd5f6-4577-4b82-8903-24648a5493e6-kube-api-access-zmjc8\") pod \"maas-api-revoke-sub-bef0d341-znp5p\" (UID: \"5eadd5f6-4577-4b82-8903-24648a5493e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef0d341-znp5p" Oct 30 00:21:02.632483 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:02.632459 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eadd5f6-4577-4b82-8903-24648a5493e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bef0d341-znp5p\" (UID: \"5eadd5f6-4577-4b82-8903-24648a5493e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef0d341-znp5p" Oct 30 00:21:02.636170 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:02.632060 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eadd5f6-4577-4b82-8903-24648a5493e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bef0d341-znp5p\" (UID: \"5eadd5f6-4577-4b82-8903-24648a5493e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef0d341-znp5p" Oct 30 00:21:02.636722 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:02.636701 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eadd5f6-4577-4b82-8903-24648a5493e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bef0d341-znp5p\" (UID: \"5eadd5f6-4577-4b82-8903-24648a5493e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef0d341-znp5p" Oct 30 00:21:02.654050 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:02.654030 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmjc8\" (UniqueName: \"kubernetes.io/projected/5eadd5f6-4577-4b82-8903-24648a5493e6-kube-api-access-zmjc8\") pod \"maas-api-revoke-sub-bef0d341-znp5p\" (UID: \"5eadd5f6-4577-4b82-8903-24648a5493e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef0d341-znp5p" Oct 30 00:21:02.858208 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:02.858166 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef0d341-znp5p" Oct 30 00:21:03.172875 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:03.172790 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" Oct 30 00:21:03.172875 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:03.172836 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" Oct 30 00:21:03.185250 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:03.185221 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" Oct 30 00:21:03.242553 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:03.242529 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bef0d341-znp5p"] Oct 30 00:21:03.455735 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:03.455696 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef0d341-znp5p" event={"ID":"5eadd5f6-4577-4b82-8903-24648a5493e6","Type":"ContainerStarted","Data":"c4251a1b61927e2432133cb8afb5ba4083e5b365e821122c16ed3f22aa27b829"} Oct 30 00:21:03.461952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:03.461901 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f86bc7-z48bk" event={"ID":"2bec19ed-2ef0-41d3-a53c-6275eea40fb9","Type":"ContainerStarted","Data":"1681f03cd503a94f2fc185b2297d30038fd2873f8250b65a6bae1b624779f9d9"} Oct 30 00:21:03.462199 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:03.462165 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f86bc7-z48bk" event={"ID":"2bec19ed-2ef0-41d3-a53c-6275eea40fb9","Type":"ContainerStarted","Data":"ea3cd3b938c38eb534571767e7bfaf646222cfc9b98dcbcb59855da5c6ed0f56"} Oct 30 00:21:03.528123 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:03.528063 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f86bc7-z48bk" podStartSLOduration=5.528040116 podStartE2EDuration="5.528040116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:03.50866223 +0000 UTC m=+978.714163411" watchObservedRunningTime="2026-10-30 00:21:03.528040116 +0000 UTC m=+978.733541297" Oct 30 00:21:03.548808 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:03.548778 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" Oct 30 00:21:04.470474 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:04.470437 5001 generic.go:356] "Generic (PLEG): container finished" podID="5eadd5f6-4577-4b82-8903-24648a5493e6" containerID="11b88c33130d0f88e2888e6f2cc6277c1b5736c4c7cc9c6d4c003fde7d5537f1" exitCode=0 Oct 30 00:21:04.470670 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:04.470524 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef0d341-znp5p" event={"ID":"5eadd5f6-4577-4b82-8903-24648a5493e6","Type":"ContainerDied","Data":"11b88c33130d0f88e2888e6f2cc6277c1b5736c4c7cc9c6d4c003fde7d5537f1"} Oct 30 00:21:04.472380 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:04.472353 5001 generic.go:356] "Generic (PLEG): container finished" podID="2bec19ed-2ef0-41d3-a53c-6275eea40fb9" containerID="1681f03cd503a94f2fc185b2297d30038fd2873f8250b65a6bae1b624779f9d9" exitCode=0 Oct 30 00:21:04.472545 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:04.472429 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f86bc7-z48bk" event={"ID":"2bec19ed-2ef0-41d3-a53c-6275eea40fb9","Type":"ContainerDied","Data":"1681f03cd503a94f2fc185b2297d30038fd2873f8250b65a6bae1b624779f9d9"} Oct 30 00:21:05.357102 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.357067 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh"] Oct 30 00:21:05.362081 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.362057 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:21:05.414560 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.414343 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh"] Oct 30 00:21:05.445770 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.445732 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:21:05.445977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.445844 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:21:05.445977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.445922 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:21:05.445977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.445963 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:21:05.446129 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.446034 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:21:05.446129 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.446091 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:21:05.446205 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.446156 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:21:05.446332 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.446315 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpzvz\" (UniqueName: \"kubernetes.io/projected/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-kube-api-access-vpzvz\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:21:05.446387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.446365 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:21:05.506012 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.505341 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e2a011-dd5bd7995-m4h7w" Oct 30 00:21:05.548708 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.548483 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:21:05.548708 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.548596 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:21:05.549590 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.549511 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpzvz\" (UniqueName: \"kubernetes.io/projected/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-kube-api-access-vpzvz\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:21:05.549762 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.549717 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:21:05.549928 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.549857 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:21:05.550495 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.550063 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:21:05.550495 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.550179 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:21:05.550495 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.550259 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:21:05.550495 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.550301 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:21:05.550495 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.550329 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:21:05.550812 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.550558 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:21:05.551060 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.551040 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:21:05.551714 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.551565 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:21:05.551714 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.551669 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:21:05.562809 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.555615 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:21:05.562809 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.562732 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:21:05.575480 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.575378 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:21:05.576081 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.576059 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpzvz\" (UniqueName: \"kubernetes.io/projected/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-kube-api-access-vpzvz\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:21:05.683527 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.682258 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:21:05.747120 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.747089 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef0d341-znp5p" Oct 30 00:21:05.750972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.748963 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59167bf4-fkjd2"] Oct 30 00:21:05.750972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.750524 5001 cpu_manager.go:414] "containerMap: removing container" podUID="5eadd5f6-4577-4b82-8903-24648a5493e6" containerName="revoke-keys" Oct 30 00:21:05.750972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.750543 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eadd5f6-4577-4b82-8903-24648a5493e6" containerName="revoke-keys" Oct 30 00:21:05.750972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.750689 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eadd5f6-4577-4b82-8903-24648a5493e6" containerName="revoke-keys" Oct 30 00:21:05.750972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.750822 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f86bc7-z48bk" Oct 30 00:21:05.755146 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.755123 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59167bf4-fkjd2" Oct 30 00:21:05.781413 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.781334 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59167bf4-fkjd2"] Oct 30 00:21:05.854830 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.854753 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bec19ed-2ef0-41d3-a53c-6275eea40fb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bec19ed-2ef0-41d3-a53c-6275eea40fb9-maas-api-xks-ca\") pod \"2bec19ed-2ef0-41d3-a53c-6275eea40fb9\" (UID: \"2bec19ed-2ef0-41d3-a53c-6275eea40fb9\") " Oct 30 00:21:05.854830 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.854812 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eadd5f6-4577-4b82-8903-24648a5493e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eadd5f6-4577-4b82-8903-24648a5493e6-maas-api-openshift-ca\") pod \"5eadd5f6-4577-4b82-8903-24648a5493e6\" (UID: \"5eadd5f6-4577-4b82-8903-24648a5493e6\") " Oct 30 00:21:05.855120 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.854906 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bec19ed-2ef0-41d3-a53c-6275eea40fb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bec19ed-2ef0-41d3-a53c-6275eea40fb9-maas-api-openshift-ca\") pod \"2bec19ed-2ef0-41d3-a53c-6275eea40fb9\" (UID: \"2bec19ed-2ef0-41d3-a53c-6275eea40fb9\") " Oct 30 00:21:05.855120 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.855026 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bec19ed-2ef0-41d3-a53c-6275eea40fb9-kube-api-access-n89bj\" (UniqueName: \"kubernetes.io/projected/2bec19ed-2ef0-41d3-a53c-6275eea40fb9-kube-api-access-n89bj\") pod \"2bec19ed-2ef0-41d3-a53c-6275eea40fb9\" (UID: \"2bec19ed-2ef0-41d3-a53c-6275eea40fb9\") " Oct 30 00:21:05.855120 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.855050 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eadd5f6-4577-4b82-8903-24648a5493e6-kube-api-access-zmjc8\" (UniqueName: \"kubernetes.io/projected/5eadd5f6-4577-4b82-8903-24648a5493e6-kube-api-access-zmjc8\") pod \"5eadd5f6-4577-4b82-8903-24648a5493e6\" (UID: \"5eadd5f6-4577-4b82-8903-24648a5493e6\") " Oct 30 00:21:05.855120 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.855069 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5eadd5f6-4577-4b82-8903-24648a5493e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eadd5f6-4577-4b82-8903-24648a5493e6-maas-api-xks-ca\") pod \"5eadd5f6-4577-4b82-8903-24648a5493e6\" (UID: \"5eadd5f6-4577-4b82-8903-24648a5493e6\") " Oct 30 00:21:05.855571 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.855549 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a2843b-681e-4663-a1ea-020c50c1d701-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59167bf4-fkjd2\" (UID: \"d5a2843b-681e-4663-a1ea-020c50c1d701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59167bf4-fkjd2" Oct 30 00:21:05.855654 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.855598 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eadd5f6-4577-4b82-8903-24648a5493e6-maas-api-openshift-ca" pod "5eadd5f6-4577-4b82-8903-24648a5493e6" (UID: "5eadd5f6-4577-4b82-8903-24648a5493e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:21:05.855654 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.855603 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bec19ed-2ef0-41d3-a53c-6275eea40fb9-maas-api-openshift-ca" pod "2bec19ed-2ef0-41d3-a53c-6275eea40fb9" (UID: "2bec19ed-2ef0-41d3-a53c-6275eea40fb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:21:05.855812 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.855784 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a2843b-681e-4663-a1ea-020c50c1d701-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59167bf4-fkjd2\" (UID: \"d5a2843b-681e-4663-a1ea-020c50c1d701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59167bf4-fkjd2" Oct 30 00:21:05.856141 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.856109 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lcbt\" (UniqueName: \"kubernetes.io/projected/d5a2843b-681e-4663-a1ea-020c50c1d701-kube-api-access-2lcbt\") pod \"maas-api-revoke-sub-59167bf4-fkjd2\" (UID: \"d5a2843b-681e-4663-a1ea-020c50c1d701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59167bf4-fkjd2" Oct 30 00:21:05.856303 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.856284 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bec19ed-2ef0-41d3-a53c-6275eea40fb9-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:21:05.856369 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.856309 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eadd5f6-4577-4b82-8903-24648a5493e6-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:21:05.857862 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.857570 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bec19ed-2ef0-41d3-a53c-6275eea40fb9-maas-api-xks-ca" pod "2bec19ed-2ef0-41d3-a53c-6275eea40fb9" (UID: "2bec19ed-2ef0-41d3-a53c-6275eea40fb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:21:05.857862 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.857656 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eadd5f6-4577-4b82-8903-24648a5493e6-maas-api-xks-ca" pod "5eadd5f6-4577-4b82-8903-24648a5493e6" (UID: "5eadd5f6-4577-4b82-8903-24648a5493e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:21:05.858333 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.858308 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bec19ed-2ef0-41d3-a53c-6275eea40fb9-kube-api-access-n89bj" pod "2bec19ed-2ef0-41d3-a53c-6275eea40fb9" (UID: "2bec19ed-2ef0-41d3-a53c-6275eea40fb9"). InnerVolumeSpecName "kube-api-access-n89bj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:21:05.858431 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.858388 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eadd5f6-4577-4b82-8903-24648a5493e6-kube-api-access-zmjc8" pod "5eadd5f6-4577-4b82-8903-24648a5493e6" (UID: "5eadd5f6-4577-4b82-8903-24648a5493e6"). InnerVolumeSpecName "kube-api-access-zmjc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:21:05.957789 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.957751 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lcbt\" (UniqueName: \"kubernetes.io/projected/d5a2843b-681e-4663-a1ea-020c50c1d701-kube-api-access-2lcbt\") pod \"maas-api-revoke-sub-59167bf4-fkjd2\" (UID: \"d5a2843b-681e-4663-a1ea-020c50c1d701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59167bf4-fkjd2" Oct 30 00:21:05.958007 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.957872 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a2843b-681e-4663-a1ea-020c50c1d701-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59167bf4-fkjd2\" (UID: \"d5a2843b-681e-4663-a1ea-020c50c1d701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59167bf4-fkjd2" Oct 30 00:21:05.958087 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.958021 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a2843b-681e-4663-a1ea-020c50c1d701-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59167bf4-fkjd2\" (UID: \"d5a2843b-681e-4663-a1ea-020c50c1d701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59167bf4-fkjd2" Oct 30 00:21:05.958306 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.958288 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n89bj\" (UniqueName: \"kubernetes.io/projected/2bec19ed-2ef0-41d3-a53c-6275eea40fb9-kube-api-access-n89bj\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:21:05.958343 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.958311 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmjc8\" (UniqueName: \"kubernetes.io/projected/5eadd5f6-4577-4b82-8903-24648a5493e6-kube-api-access-zmjc8\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:21:05.958343 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.958326 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eadd5f6-4577-4b82-8903-24648a5493e6-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:21:05.958407 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.958340 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bec19ed-2ef0-41d3-a53c-6275eea40fb9-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:21:05.958586 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.958564 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a2843b-681e-4663-a1ea-020c50c1d701-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59167bf4-fkjd2\" (UID: \"d5a2843b-681e-4663-a1ea-020c50c1d701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59167bf4-fkjd2" Oct 30 00:21:05.960690 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.960663 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a2843b-681e-4663-a1ea-020c50c1d701-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59167bf4-fkjd2\" (UID: \"d5a2843b-681e-4663-a1ea-020c50c1d701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59167bf4-fkjd2" Oct 30 00:21:05.978489 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:05.978460 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lcbt\" (UniqueName: \"kubernetes.io/projected/d5a2843b-681e-4663-a1ea-020c50c1d701-kube-api-access-2lcbt\") pod \"maas-api-revoke-sub-59167bf4-fkjd2\" (UID: \"d5a2843b-681e-4663-a1ea-020c50c1d701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59167bf4-fkjd2" Oct 30 00:21:06.029329 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:21:06.029297 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b9e2b08_e5d3_4ad2_85fd_eda8e00d9f86.slice/crio-bece2efe3c0f88f83a5afcb317f22ae88098bf0b9c3db0cd25161f5b19ec27af WatchSource:0}: Error finding container bece2efe3c0f88f83a5afcb317f22ae88098bf0b9c3db0cd25161f5b19ec27af: Status 404 returned error can't find the container with id bece2efe3c0f88f83a5afcb317f22ae88098bf0b9c3db0cd25161f5b19ec27af Oct 30 00:21:06.029584 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:06.029562 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh"] Oct 30 00:21:06.032118 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:06.032089 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:21:06.032187 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:06.032144 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:21:06.032231 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:06.032197 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:21:06.084278 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:06.083978 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59167bf4-fkjd2" Oct 30 00:21:06.401795 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:06.401711 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59167bf4-fkjd2"] Oct 30 00:21:06.409743 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:21:06.409710 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a2843b_681e_4663_a1ea_020c50c1d701.slice/crio-6fa71864be96a92b24f9c3b0283b3e762332be55571a26a4913a0eabdbdb9955 WatchSource:0}: Error finding container 6fa71864be96a92b24f9c3b0283b3e762332be55571a26a4913a0eabdbdb9955: Status 404 returned error can't find the container with id 6fa71864be96a92b24f9c3b0283b3e762332be55571a26a4913a0eabdbdb9955 Oct 30 00:21:06.549196 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:06.549104 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59167bf4-fkjd2" event={"ID":"d5a2843b-681e-4663-a1ea-020c50c1d701","Type":"ContainerStarted","Data":"6fa71864be96a92b24f9c3b0283b3e762332be55571a26a4913a0eabdbdb9955"} Oct 30 00:21:06.580031 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:06.558994 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef0d341-znp5p" event={"ID":"5eadd5f6-4577-4b82-8903-24648a5493e6","Type":"ContainerDied","Data":"c4251a1b61927e2432133cb8afb5ba4083e5b365e821122c16ed3f22aa27b829"} Oct 30 00:21:06.580031 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:06.559064 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4251a1b61927e2432133cb8afb5ba4083e5b365e821122c16ed3f22aa27b829" Oct 30 00:21:06.580031 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:06.559136 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef0d341-znp5p" Oct 30 00:21:06.619754 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:06.618936 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" event={"ID":"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86","Type":"ContainerStarted","Data":"08757b8b027232ff366bb669a20944beba02ea04ff260ecb3a73af46602e043c"} Oct 30 00:21:06.619754 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:06.618990 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" event={"ID":"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86","Type":"ContainerStarted","Data":"bece2efe3c0f88f83a5afcb317f22ae88098bf0b9c3db0cd25161f5b19ec27af"} Oct 30 00:21:06.622736 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:06.622709 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f86bc7-z48bk" Oct 30 00:21:06.623285 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:06.622716 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f86bc7-z48bk" event={"ID":"2bec19ed-2ef0-41d3-a53c-6275eea40fb9","Type":"ContainerDied","Data":"ea3cd3b938c38eb534571767e7bfaf646222cfc9b98dcbcb59855da5c6ed0f56"} Oct 30 00:21:06.623390 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:06.623379 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea3cd3b938c38eb534571767e7bfaf646222cfc9b98dcbcb59855da5c6ed0f56" Oct 30 00:21:06.660419 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:06.660314 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" podStartSLOduration=1.6602963640000001 podStartE2EDuration="1.660296364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:06.659053941 +0000 UTC m=+981.864555123" watchObservedRunningTime="2026-10-30 00:21:06.660296364 +0000 UTC m=+981.865797547" Oct 30 00:21:06.683327 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:06.683296 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:21:07.643707 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:07.643366 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59167bf4-fkjd2" event={"ID":"d5a2843b-681e-4663-a1ea-020c50c1d701","Type":"ContainerStarted","Data":"c27e5730599c7cd23bffab855204bad71774a5b6f68d86462fb3a76b35cf5290"} Oct 30 00:21:07.680513 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:07.680254 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59167bf4-fkjd2" podStartSLOduration=2.680234454 podStartE2EDuration="2.680234454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:07.677889058 +0000 UTC m=+982.883390277" watchObservedRunningTime="2026-10-30 00:21:07.680234454 +0000 UTC m=+982.885735636" Oct 30 00:21:07.685009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:07.684969 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 00:21:07.685170 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:07.685032 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" podUID="6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 00:21:08.665967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:08.663840 5001 generic.go:356] "Generic (PLEG): container finished" podID="d5a2843b-681e-4663-a1ea-020c50c1d701" containerID="c27e5730599c7cd23bffab855204bad71774a5b6f68d86462fb3a76b35cf5290" exitCode=0 Oct 30 00:21:08.665967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:08.665416 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59167bf4-fkjd2" event={"ID":"d5a2843b-681e-4663-a1ea-020c50c1d701","Type":"ContainerDied","Data":"c27e5730599c7cd23bffab855204bad71774a5b6f68d86462fb3a76b35cf5290"} Oct 30 00:21:08.683632 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:08.683596 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 00:21:08.683818 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:08.683657 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" podUID="6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 00:21:09.163935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.160021 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2"] Oct 30 00:21:09.163935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.161117 5001 cpu_manager.go:414] "containerMap: removing container" podUID="2bec19ed-2ef0-41d3-a53c-6275eea40fb9" containerName="revoke-keys" Oct 30 00:21:09.163935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.161139 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bec19ed-2ef0-41d3-a53c-6275eea40fb9" containerName="revoke-keys" Oct 30 00:21:09.163935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.161366 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bec19ed-2ef0-41d3-a53c-6275eea40fb9" containerName="revoke-keys" Oct 30 00:21:09.192930 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.187253 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:21:09.192930 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.191173 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2"] Oct 30 00:21:09.345240 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.345192 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:21:09.345434 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.345278 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:21:09.345434 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.345312 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00301bd4-1afd-4dd3-8fbc-45376008859e-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:21:09.345564 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.345449 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxxqs\" (UniqueName: \"kubernetes.io/projected/00301bd4-1afd-4dd3-8fbc-45376008859e-kube-api-access-nxxqs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:21:09.348947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.345671 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:21:09.348947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.345701 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00301bd4-1afd-4dd3-8fbc-45376008859e-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:21:09.348947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.345770 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:21:09.348947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.345796 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:21:09.348947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.345904 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00301bd4-1afd-4dd3-8fbc-45376008859e-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:21:09.456965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.456364 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:21:09.456965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.456410 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00301bd4-1afd-4dd3-8fbc-45376008859e-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:21:09.456965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.456468 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:21:09.456965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.456492 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:21:09.456965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.456591 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00301bd4-1afd-4dd3-8fbc-45376008859e-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:21:09.456965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.456632 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:21:09.456965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.456676 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:21:09.456965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.456707 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00301bd4-1afd-4dd3-8fbc-45376008859e-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:21:09.456965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.456838 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxxqs\" (UniqueName: \"kubernetes.io/projected/00301bd4-1afd-4dd3-8fbc-45376008859e-kube-api-access-nxxqs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:21:09.458414 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.458378 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:21:09.479870 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.479811 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:21:09.479870 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.479853 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00301bd4-1afd-4dd3-8fbc-45376008859e-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:21:09.480153 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.480009 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:21:09.480247 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.480228 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:21:09.519710 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.515963 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn"] Oct 30 00:21:09.529020 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.528450 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn"] Oct 30 00:21:09.529020 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.528568 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:21:09.553934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.552904 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxxqs\" (UniqueName: \"kubernetes.io/projected/00301bd4-1afd-4dd3-8fbc-45376008859e-kube-api-access-nxxqs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:21:09.553934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.553484 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00301bd4-1afd-4dd3-8fbc-45376008859e-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:21:09.554529 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.554501 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:21:09.555861 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.555837 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00301bd4-1afd-4dd3-8fbc-45376008859e-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:21:09.683105 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.683073 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 00:21:09.683565 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.683133 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" podUID="6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 00:21:09.686190 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.686158 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:21:09.686377 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.686356 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:21:09.686459 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.686410 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c596e5d-d36f-4543-836e-15ee5d3b111b-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:21:09.686533 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.686474 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:21:09.686533 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.686513 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:21:09.686650 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.686538 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c596e5d-d36f-4543-836e-15ee5d3b111b-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:21:09.686706 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.686685 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:21:09.686809 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.686790 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fntfg\" (UniqueName: \"kubernetes.io/projected/8c596e5d-d36f-4543-836e-15ee5d3b111b-kube-api-access-fntfg\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:21:09.686870 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.686831 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c596e5d-d36f-4543-836e-15ee5d3b111b-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:21:09.697376 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.697351 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:21:09.792482 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.788390 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c596e5d-d36f-4543-836e-15ee5d3b111b-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:21:09.792482 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.788512 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:21:09.792482 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.788577 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fntfg\" (UniqueName: \"kubernetes.io/projected/8c596e5d-d36f-4543-836e-15ee5d3b111b-kube-api-access-fntfg\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:21:09.792482 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.788623 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c596e5d-d36f-4543-836e-15ee5d3b111b-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:21:09.792482 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.788772 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:21:09.792482 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.788956 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:21:09.792482 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.789006 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c596e5d-d36f-4543-836e-15ee5d3b111b-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:21:09.792482 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.789042 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:21:09.792482 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.789075 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:21:09.792482 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.789540 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:21:09.792482 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.790185 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:21:09.792482 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.791746 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c596e5d-d36f-4543-836e-15ee5d3b111b-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:21:09.792482 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.792029 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:21:09.792482 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.792077 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:21:09.794753 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.794707 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:21:09.795701 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.795674 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c596e5d-d36f-4543-836e-15ee5d3b111b-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:21:09.909279 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.908680 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c596e5d-d36f-4543-836e-15ee5d3b111b-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:21:09.971739 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:09.971662 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fntfg\" (UniqueName: \"kubernetes.io/projected/8c596e5d-d36f-4543-836e-15ee5d3b111b-kube-api-access-fntfg\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:21:10.237104 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:10.236937 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:21:10.244343 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:10.243678 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59167bf4-fkjd2" Oct 30 00:21:10.352612 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:10.351901 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2"] Oct 30 00:21:10.358178 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:21:10.358134 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00301bd4_1afd_4dd3_8fbc_45376008859e.slice/crio-a238a5bfd17338d00e992b770a7777903378a0eb12e8fb2cb5b1a5876ca03481 WatchSource:0}: Error finding container a238a5bfd17338d00e992b770a7777903378a0eb12e8fb2cb5b1a5876ca03481: Status 404 returned error can't find the container with id a238a5bfd17338d00e992b770a7777903378a0eb12e8fb2cb5b1a5876ca03481 Oct 30 00:21:10.360803 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:10.360766 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:21:10.360926 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:10.360849 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:21:10.360926 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:10.360896 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:21:10.397407 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:10.397076 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5a2843b-681e-4663-a1ea-020c50c1d701-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a2843b-681e-4663-a1ea-020c50c1d701-maas-api-xks-ca\") pod \"d5a2843b-681e-4663-a1ea-020c50c1d701\" (UID: \"d5a2843b-681e-4663-a1ea-020c50c1d701\") " Oct 30 00:21:10.398029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:10.397610 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5a2843b-681e-4663-a1ea-020c50c1d701-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a2843b-681e-4663-a1ea-020c50c1d701-maas-api-openshift-ca\") pod \"d5a2843b-681e-4663-a1ea-020c50c1d701\" (UID: \"d5a2843b-681e-4663-a1ea-020c50c1d701\") " Oct 30 00:21:10.398029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:10.397763 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a2843b-681e-4663-a1ea-020c50c1d701-kube-api-access-2lcbt\" (UniqueName: \"kubernetes.io/projected/d5a2843b-681e-4663-a1ea-020c50c1d701-kube-api-access-2lcbt\") pod \"d5a2843b-681e-4663-a1ea-020c50c1d701\" (UID: \"d5a2843b-681e-4663-a1ea-020c50c1d701\") " Oct 30 00:21:10.398029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:10.398006 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a2843b-681e-4663-a1ea-020c50c1d701-maas-api-openshift-ca" pod "d5a2843b-681e-4663-a1ea-020c50c1d701" (UID: "d5a2843b-681e-4663-a1ea-020c50c1d701"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:21:10.398619 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:10.398570 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a2843b-681e-4663-a1ea-020c50c1d701-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:21:10.405521 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:10.405492 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5a2843b-681e-4663-a1ea-020c50c1d701-maas-api-xks-ca" pod "d5a2843b-681e-4663-a1ea-020c50c1d701" (UID: "d5a2843b-681e-4663-a1ea-020c50c1d701"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:21:10.406462 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:10.406421 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a2843b-681e-4663-a1ea-020c50c1d701-kube-api-access-2lcbt" pod "d5a2843b-681e-4663-a1ea-020c50c1d701" (UID: "d5a2843b-681e-4663-a1ea-020c50c1d701"). InnerVolumeSpecName "kube-api-access-2lcbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:21:10.504237 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:10.504202 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lcbt\" (UniqueName: \"kubernetes.io/projected/d5a2843b-681e-4663-a1ea-020c50c1d701-kube-api-access-2lcbt\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:21:10.504237 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:10.504239 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a2843b-681e-4663-a1ea-020c50c1d701-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:21:10.658974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:10.658490 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn"] Oct 30 00:21:10.659644 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:21:10.659344 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c596e5d_d36f_4543_836e_15ee5d3b111b.slice/crio-961e9b45c8dc7a670c8f644bffd674705a23cadb968d76da7b1d2e2743f41d7e WatchSource:0}: Error finding container 961e9b45c8dc7a670c8f644bffd674705a23cadb968d76da7b1d2e2743f41d7e: Status 404 returned error can't find the container with id 961e9b45c8dc7a670c8f644bffd674705a23cadb968d76da7b1d2e2743f41d7e Oct 30 00:21:10.662607 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:10.662574 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:21:10.662694 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:10.662658 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:21:10.662771 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:10.662694 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:21:10.683089 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:10.683042 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 00:21:10.683244 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:10.683098 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" podUID="6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 00:21:10.686403 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:10.686338 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" event={"ID":"8c596e5d-d36f-4543-836e-15ee5d3b111b","Type":"ContainerStarted","Data":"961e9b45c8dc7a670c8f644bffd674705a23cadb968d76da7b1d2e2743f41d7e"} Oct 30 00:21:10.688794 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:10.688767 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" event={"ID":"00301bd4-1afd-4dd3-8fbc-45376008859e","Type":"ContainerStarted","Data":"95aa6314df85021ded6bef94b0eb2dcfafa4905540c3906fb1620cf13d953503"} Oct 30 00:21:10.688921 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:10.688820 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" event={"ID":"00301bd4-1afd-4dd3-8fbc-45376008859e","Type":"ContainerStarted","Data":"a238a5bfd17338d00e992b770a7777903378a0eb12e8fb2cb5b1a5876ca03481"} Oct 30 00:21:10.690558 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:10.690533 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59167bf4-fkjd2" event={"ID":"d5a2843b-681e-4663-a1ea-020c50c1d701","Type":"ContainerDied","Data":"6fa71864be96a92b24f9c3b0283b3e762332be55571a26a4913a0eabdbdb9955"} Oct 30 00:21:10.690669 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:10.690580 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fa71864be96a92b24f9c3b0283b3e762332be55571a26a4913a0eabdbdb9955" Oct 30 00:21:10.690669 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:10.690641 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59167bf4-fkjd2" Oct 30 00:21:10.698376 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:10.698350 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:21:10.747011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:10.743750 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" podStartSLOduration=1.7437278630000002 podStartE2EDuration="1.743727863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:10.730213584 +0000 UTC m=+985.935714766" watchObservedRunningTime="2026-10-30 00:21:10.743727863 +0000 UTC m=+985.949229045" Oct 30 00:21:11.053631 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:11.053561 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a51b204e-4g5zc"] Oct 30 00:21:11.057686 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:11.057656 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a51b204e-4g5zc"] Oct 30 00:21:11.683556 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:11.683065 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 00:21:11.683556 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:11.683129 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" podUID="6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 00:21:11.701827 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:11.701695 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 00:21:11.701827 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:11.701757 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" podUID="00301bd4-1afd-4dd3-8fbc-45376008859e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 00:21:11.772109 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:11.772027 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" event={"ID":"8c596e5d-d36f-4543-836e-15ee5d3b111b","Type":"ContainerStarted","Data":"376c865cfccf8e75227c21f756202b3cfbed20b6950d75aa58a781a1093a6d36"} Oct 30 00:21:12.006841 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:12.006716 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" podStartSLOduration=3.006692193 podStartE2EDuration="3.006692193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:11.840416402 +0000 UTC m=+987.045917585" watchObservedRunningTime="2026-10-30 00:21:12.006692193 +0000 UTC m=+987.212193375" Oct 30 00:21:12.398936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:12.375444 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:21:12.451933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:12.451795 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 00:21:12.451933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:12.451900 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" podUID="8c596e5d-d36f-4543-836e-15ee5d3b111b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 00:21:12.699933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:12.696850 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 00:21:12.699933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:12.696906 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" podUID="6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 00:21:12.699933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:12.697621 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5ad099d-2698c"] Oct 30 00:21:12.699933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:12.698617 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 00:21:12.699933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:12.698664 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" podUID="00301bd4-1afd-4dd3-8fbc-45376008859e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 00:21:12.705561 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:12.705532 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5ad099d-2698c"] Oct 30 00:21:13.290607 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:13.290573 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 00:21:13.291080 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:13.290957 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" podUID="8c596e5d-d36f-4543-836e-15ee5d3b111b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 00:21:13.433684 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:13.433637 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af1b6516-758d-4a1d-a89b-0198c9d3ad46" path="/var/lib/kubelet/pods/af1b6516-758d-4a1d-a89b-0198c9d3ad46/volumes" Oct 30 00:21:13.471972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:13.469878 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d57fa402-3023-47ed-b4ff-2b90737d422c" path="/var/lib/kubelet/pods/d57fa402-3023-47ed-b4ff-2b90737d422c/volumes" Oct 30 00:21:13.471972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:13.470442 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0751af9f-vrtt9"] Oct 30 00:21:13.471972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:13.471425 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d5a2843b-681e-4663-a1ea-020c50c1d701" containerName="revoke-keys" Oct 30 00:21:13.471972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:13.471444 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a2843b-681e-4663-a1ea-020c50c1d701" containerName="revoke-keys" Oct 30 00:21:13.471972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:13.471668 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a2843b-681e-4663-a1ea-020c50c1d701" containerName="revoke-keys" Oct 30 00:21:13.551169 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:13.518730 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0751af9f-vrtt9"] Oct 30 00:21:13.551169 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:13.518850 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0751af9f-vrtt9" Oct 30 00:21:13.561405 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:13.554787 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:21:13.584725 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:13.567271 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:21:13.584725 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:13.567388 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:21:13.584725 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:13.567458 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:21:13.669029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:13.667257 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhmcn\" (UniqueName: \"kubernetes.io/projected/102fe425-f28e-4191-881e-369a6b35a761-kube-api-access-rhmcn\") pod \"maas-api-revoke-sub-0751af9f-vrtt9\" (UID: \"102fe425-f28e-4191-881e-369a6b35a761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0751af9f-vrtt9" Oct 30 00:21:13.669029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:13.667375 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/102fe425-f28e-4191-881e-369a6b35a761-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0751af9f-vrtt9\" (UID: \"102fe425-f28e-4191-881e-369a6b35a761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0751af9f-vrtt9" Oct 30 00:21:13.669029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:13.667475 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/102fe425-f28e-4191-881e-369a6b35a761-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0751af9f-vrtt9\" (UID: \"102fe425-f28e-4191-881e-369a6b35a761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0751af9f-vrtt9" Oct 30 00:21:13.690737 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:13.690702 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 00:21:13.691050 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:13.691020 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" podUID="6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 00:21:13.699363 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:13.699330 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 00:21:13.699607 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:13.699579 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" podUID="00301bd4-1afd-4dd3-8fbc-45376008859e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 00:21:13.771185 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:13.771143 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhmcn\" (UniqueName: \"kubernetes.io/projected/102fe425-f28e-4191-881e-369a6b35a761-kube-api-access-rhmcn\") pod \"maas-api-revoke-sub-0751af9f-vrtt9\" (UID: \"102fe425-f28e-4191-881e-369a6b35a761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0751af9f-vrtt9" Oct 30 00:21:13.810077 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:13.772169 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/102fe425-f28e-4191-881e-369a6b35a761-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0751af9f-vrtt9\" (UID: \"102fe425-f28e-4191-881e-369a6b35a761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0751af9f-vrtt9" Oct 30 00:21:13.810077 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:13.772295 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/102fe425-f28e-4191-881e-369a6b35a761-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0751af9f-vrtt9\" (UID: \"102fe425-f28e-4191-881e-369a6b35a761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0751af9f-vrtt9" Oct 30 00:21:13.810077 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:13.773512 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/102fe425-f28e-4191-881e-369a6b35a761-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0751af9f-vrtt9\" (UID: \"102fe425-f28e-4191-881e-369a6b35a761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0751af9f-vrtt9" Oct 30 00:21:13.816637 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:13.816564 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/102fe425-f28e-4191-881e-369a6b35a761-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0751af9f-vrtt9\" (UID: \"102fe425-f28e-4191-881e-369a6b35a761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0751af9f-vrtt9" Oct 30 00:21:13.914937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:13.898670 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhmcn\" (UniqueName: \"kubernetes.io/projected/102fe425-f28e-4191-881e-369a6b35a761-kube-api-access-rhmcn\") pod \"maas-api-revoke-sub-0751af9f-vrtt9\" (UID: \"102fe425-f28e-4191-881e-369a6b35a761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0751af9f-vrtt9" Oct 30 00:21:13.930441 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:13.930151 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0751af9f-vrtt9" Oct 30 00:21:14.246016 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:14.238311 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 00:21:14.246016 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:14.238367 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" podUID="8c596e5d-d36f-4543-836e-15ee5d3b111b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 00:21:14.697853 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:14.697820 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 00:21:14.698290 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:14.697880 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" podUID="00301bd4-1afd-4dd3-8fbc-45376008859e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 00:21:14.722093 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:14.716152 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0751af9f-vrtt9"] Oct 30 00:21:15.048177 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:15.048084 5001 generic.go:356] "Generic (PLEG): container finished" podID="f42670c1-7716-46d8-9bc3-3f06cf8caa8a" containerID="5228870d920b4c45c868cb243140a6e055def8b47c6d8277afa434d652ba9250" exitCode=0 Oct 30 00:21:15.074212 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:15.074178 5001 patch_prober.go:29] interesting pod/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 00:21:15.087969 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:15.087928 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" podUID="f42670c1-7716-46d8-9bc3-3f06cf8caa8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 00:21:15.103267 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:15.103229 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" event={"ID":"f42670c1-7716-46d8-9bc3-3f06cf8caa8a","Type":"ContainerDied","Data":"5228870d920b4c45c868cb243140a6e055def8b47c6d8277afa434d652ba9250"} Oct 30 00:21:15.103501 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:15.103481 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0751af9f-vrtt9" event={"ID":"102fe425-f28e-4191-881e-369a6b35a761","Type":"ContainerStarted","Data":"2eacd7109bf23d4dd6dabab22d8fb7fbc157898df0266dc23d19103cb1f9b370"} Oct 30 00:21:15.104099 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:15.104064 5001 scope.go:122] "RemoveContainer" containerID="5228870d920b4c45c868cb243140a6e055def8b47c6d8277afa434d652ba9250" Oct 30 00:21:15.106471 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:15.106411 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:21:15.106562 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:15.106502 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:21:15.106602 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:15.106562 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:21:15.270135 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:15.252193 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 00:21:15.270135 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:15.252249 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" podUID="8c596e5d-d36f-4543-836e-15ee5d3b111b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 00:21:15.433945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:15.432369 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:21:15.443103 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:15.439585 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:21:15.477575 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:15.461190 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:21:15.697733 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:15.697687 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 00:21:15.697957 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:15.697745 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" podUID="00301bd4-1afd-4dd3-8fbc-45376008859e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 00:21:16.200468 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:16.199216 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" event={"ID":"f42670c1-7716-46d8-9bc3-3f06cf8caa8a","Type":"ContainerStarted","Data":"e7ba467a236ec1b1538f4ad87076753316038898b6856dbdcb97803816025c94"} Oct 30 00:21:16.202297 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:16.202271 5001 generic.go:356] "Generic (PLEG): container finished" podID="102fe425-f28e-4191-881e-369a6b35a761" containerID="cd530465f37e4e538e7e49642ac86280adad3148c3a1f677e070274f9dce9dd1" exitCode=0 Oct 30 00:21:16.202439 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:16.202341 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0751af9f-vrtt9" event={"ID":"102fe425-f28e-4191-881e-369a6b35a761","Type":"ContainerDied","Data":"cd530465f37e4e538e7e49642ac86280adad3148c3a1f677e070274f9dce9dd1"} Oct 30 00:21:16.263181 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:16.261005 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 00:21:16.263181 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:16.261061 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" podUID="8c596e5d-d36f-4543-836e-15ee5d3b111b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 00:21:16.302137 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:16.302082 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:21:16.302335 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:16.302185 5001 patch_prober.go:29] interesting pod/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 00:21:16.302335 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:16.302223 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" podUID="f42670c1-7716-46d8-9bc3-3f06cf8caa8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 00:21:16.843685 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:16.843645 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 00:21:16.844023 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:16.843992 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" podUID="00301bd4-1afd-4dd3-8fbc-45376008859e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 00:21:17.289991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:17.268365 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 00:21:17.289991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:17.268424 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" podUID="8c596e5d-d36f-4543-836e-15ee5d3b111b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 00:21:17.302056 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:17.302020 5001 patch_prober.go:29] interesting pod/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 00:21:17.302222 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:17.302090 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" podUID="f42670c1-7716-46d8-9bc3-3f06cf8caa8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 00:21:17.397520 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:21:17.397470 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:21:17.710998 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:17.699452 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 00:21:17.710998 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:17.699527 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" podUID="00301bd4-1afd-4dd3-8fbc-45376008859e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 00:21:18.250028 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:18.249493 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 00:21:18.250028 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:18.249552 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" podUID="8c596e5d-d36f-4543-836e-15ee5d3b111b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 00:21:18.258147 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:18.258119 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0751af9f-vrtt9" Oct 30 00:21:18.321017 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:18.320353 5001 patch_prober.go:29] interesting pod/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 00:21:18.321017 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:18.320407 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" podUID="f42670c1-7716-46d8-9bc3-3f06cf8caa8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 00:21:18.399200 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:18.398980 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/102fe425-f28e-4191-881e-369a6b35a761-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/102fe425-f28e-4191-881e-369a6b35a761-maas-api-xks-ca\") pod \"102fe425-f28e-4191-881e-369a6b35a761\" (UID: \"102fe425-f28e-4191-881e-369a6b35a761\") " Oct 30 00:21:18.399391 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:18.399208 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/102fe425-f28e-4191-881e-369a6b35a761-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/102fe425-f28e-4191-881e-369a6b35a761-maas-api-openshift-ca\") pod \"102fe425-f28e-4191-881e-369a6b35a761\" (UID: \"102fe425-f28e-4191-881e-369a6b35a761\") " Oct 30 00:21:18.399391 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:18.399297 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/102fe425-f28e-4191-881e-369a6b35a761-kube-api-access-rhmcn\" (UniqueName: \"kubernetes.io/projected/102fe425-f28e-4191-881e-369a6b35a761-kube-api-access-rhmcn\") pod \"102fe425-f28e-4191-881e-369a6b35a761\" (UID: \"102fe425-f28e-4191-881e-369a6b35a761\") " Oct 30 00:21:18.400762 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:18.400531 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/102fe425-f28e-4191-881e-369a6b35a761-maas-api-openshift-ca" pod "102fe425-f28e-4191-881e-369a6b35a761" (UID: "102fe425-f28e-4191-881e-369a6b35a761"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:21:18.403851 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:18.403797 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/102fe425-f28e-4191-881e-369a6b35a761-maas-api-xks-ca" pod "102fe425-f28e-4191-881e-369a6b35a761" (UID: "102fe425-f28e-4191-881e-369a6b35a761"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:21:18.403997 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:18.403889 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0751af9f-vrtt9" Oct 30 00:21:18.404223 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:18.404094 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0751af9f-vrtt9" event={"ID":"102fe425-f28e-4191-881e-369a6b35a761","Type":"ContainerDied","Data":"2eacd7109bf23d4dd6dabab22d8fb7fbc157898df0266dc23d19103cb1f9b370"} Oct 30 00:21:18.404223 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:18.404198 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2eacd7109bf23d4dd6dabab22d8fb7fbc157898df0266dc23d19103cb1f9b370" Oct 30 00:21:18.448996 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:18.448294 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/102fe425-f28e-4191-881e-369a6b35a761-kube-api-access-rhmcn" pod "102fe425-f28e-4191-881e-369a6b35a761" (UID: "102fe425-f28e-4191-881e-369a6b35a761"). InnerVolumeSpecName "kube-api-access-rhmcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:21:18.502260 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:18.502218 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/102fe425-f28e-4191-881e-369a6b35a761-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:21:18.502260 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:18.502261 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/102fe425-f28e-4191-881e-369a6b35a761-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:21:18.502465 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:18.502279 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhmcn\" (UniqueName: \"kubernetes.io/projected/102fe425-f28e-4191-881e-369a6b35a761-kube-api-access-rhmcn\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:21:18.697887 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:18.697780 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 00:21:18.697887 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:18.697864 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" podUID="00301bd4-1afd-4dd3-8fbc-45376008859e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 00:21:19.244936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:19.239353 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 00:21:19.244936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:19.239414 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" podUID="8c596e5d-d36f-4543-836e-15ee5d3b111b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 00:21:19.313935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:19.313287 5001 patch_prober.go:29] interesting pod/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 00:21:19.313935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:19.313354 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" podUID="f42670c1-7716-46d8-9bc3-3f06cf8caa8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 00:21:20.189334 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:20.189297 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:21:20.190960 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:20.190149 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:21:20.191252 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:20.191231 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:21:20.327930 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:20.319135 5001 patch_prober.go:29] interesting pod/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 00:21:20.327930 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:20.319192 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" podUID="f42670c1-7716-46d8-9bc3-3f06cf8caa8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 00:21:20.372951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:20.367851 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:21:20.372951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:20.369610 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:21:20.395936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:20.389647 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:21:21.304163 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:21.304122 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 00:21:21.311238 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:21.305416 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:21.343046 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:21.338139 5001 patch_prober.go:29] interesting pod/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 00:21:21.343046 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:21.338233 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" podUID="f42670c1-7716-46d8-9bc3-3f06cf8caa8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 00:21:21.477055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:21.477000 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" podUID="b3e66631-9562-4bb5-b005-1fc179a8582d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.135:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:21.801945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:21.795229 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:22.268116 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:22.268063 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:22.308491 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:22.308461 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 00:21:22.309227 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:22.309204 5001 patch_prober.go:29] interesting pod/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 00:21:22.309465 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:22.309444 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" podUID="f42670c1-7716-46d8-9bc3-3f06cf8caa8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 00:21:22.309846 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:22.309830 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:21:22.311489 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:22.308412 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:22.312776 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:22.312073 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:22.313081 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:22.313064 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:21:22.474390 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:22.474345 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 00:21:22.525991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:22.520509 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" podUID="b3e66631-9562-4bb5-b005-1fc179a8582d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.135:8000/ready\": context deadline exceeded" Oct 30 00:21:22.525991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:22.520904 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" Oct 30 00:21:23.006995 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:22.847853 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:23.329284 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:23.329234 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:23.336419 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:23.336383 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:23.338116 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:23.338088 5001 patch_prober.go:29] interesting pod/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 00:21:23.338357 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:23.338331 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" podUID="f42670c1-7716-46d8-9bc3-3f06cf8caa8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 00:21:23.359546 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:23.359513 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:21:23.380935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:23.359969 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:21:23.385939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:23.385264 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:23.433934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:23.415013 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:23.459936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:23.456490 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:23.506955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:23.504711 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:23.506955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:23.505929 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:21:23.508701 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:23.507727 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:21:23.536937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:23.530951 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" podUID="b3e66631-9562-4bb5-b005-1fc179a8582d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.135:8000/ready\": context deadline exceeded" Oct 30 00:21:23.660997 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:23.648709 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:21:24.012704 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:24.012649 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:24.177183 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:24.177091 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" podUID="b3e66631-9562-4bb5-b005-1fc179a8582d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.135:8000/health\": context deadline exceeded" Oct 30 00:21:24.177562 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:24.177437 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:24.179259 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:24.177884 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:21:24.358996 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:24.336973 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:24.358996 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:24.338291 5001 patch_prober.go:29] interesting pod/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 00:21:24.358996 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:24.338349 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" podUID="f42670c1-7716-46d8-9bc3-3f06cf8caa8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 00:21:24.366998 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:24.359795 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:24.386994 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:24.386950 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:24.513468 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:24.486422 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:24.534888 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:24.507132 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:24.534888 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:24.518665 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:21:24.534888 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:24.518727 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:21:24.534888 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:24.507171 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:24.564603 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:24.564550 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" podUID="b3e66631-9562-4bb5-b005-1fc179a8582d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.135:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:24.663347 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:24.663264 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:21:25.340085 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:25.340041 5001 patch_prober.go:29] interesting pod/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 00:21:25.340406 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:25.340368 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" podUID="f42670c1-7716-46d8-9bc3-3f06cf8caa8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 00:21:25.366937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:25.344264 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:25.366937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:25.364058 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:25.410936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:25.393034 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:25.518406 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:25.518367 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:25.518884 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:25.518851 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:21:25.597218 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:25.521547 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:25.597985 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:25.597939 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" podUID="b3e66631-9562-4bb5-b005-1fc179a8582d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.135:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:25.873008 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:25.839697 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 00:21:26.229624 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:26.220184 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:21:26.312662 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:26.302593 5001 patch_prober.go:29] interesting pod/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 00:21:26.312662 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:26.302651 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" podUID="f42670c1-7716-46d8-9bc3-3f06cf8caa8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 00:21:26.485312 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:26.483011 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:26.485312 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:26.483813 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 30 00:21:26.544217 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:26.544172 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:26.629933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:26.614106 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 00:21:26.632996 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:26.631719 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:21:26.705281 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:26.705123 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" podUID="b3e66631-9562-4bb5-b005-1fc179a8582d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.135:8000/ready\": context deadline exceeded" Oct 30 00:21:26.780939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:26.772223 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:26.854938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:26.848777 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:26.854938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:26.849841 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:21:27.211520 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.211036 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:27.329724 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.308880 5001 patch_prober.go:29] interesting pod/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 00:21:27.329724 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.308959 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" podUID="f42670c1-7716-46d8-9bc3-3f06cf8caa8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 00:21:27.422965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.421235 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s"] Oct 30 00:21:27.446933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.443368 5001 cpu_manager.go:414] "containerMap: removing container" podUID="102fe425-f28e-4191-881e-369a6b35a761" containerName="revoke-keys" Oct 30 00:21:27.446933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.443506 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="102fe425-f28e-4191-881e-369a6b35a761" containerName="revoke-keys" Oct 30 00:21:27.472993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.469377 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="102fe425-f28e-4191-881e-369a6b35a761" containerName="revoke-keys" Oct 30 00:21:27.503936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.493096 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:27.503936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.493544 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:21:27.503936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.502454 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:21:27.565946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.564847 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 00:21:27.574942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.573833 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:21:27.574942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.573894 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8t9n\" (UniqueName: \"kubernetes.io/projected/257eedd7-4d1b-4e1f-91c5-b5af649dc927-kube-api-access-f8t9n\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:21:27.574942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.574025 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:21:27.574942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.574067 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:21:27.574942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.574120 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:21:27.574942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.574168 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:21:27.574942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.574269 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:21:27.574942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.574363 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:21:27.574942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.574406 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:21:27.624052 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.624005 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:27.677938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.649969 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd"] Oct 30 00:21:27.701686 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.700048 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:27.703664 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.703638 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:21:27.717508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.704554 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:21:27.728987 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.719199 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:21:27.733062 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.720731 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:21:27.733351 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.733326 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:21:27.733571 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.733555 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:21:27.733730 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.733717 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:21:27.733837 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.733824 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:21:27.734343 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.734320 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:21:27.734722 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.734702 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:21:27.734848 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.734831 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8t9n\" (UniqueName: \"kubernetes.io/projected/257eedd7-4d1b-4e1f-91c5-b5af649dc927-kube-api-access-f8t9n\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:21:27.735053 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.735039 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:21:27.761008 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.760648 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s"] Oct 30 00:21:27.763029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.761173 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:21:27.763029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.761482 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:21:27.763029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.761840 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:21:27.764670 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.764642 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" podUID="b3e66631-9562-4bb5-b005-1fc179a8582d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.135:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:27.774079 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.774035 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:27.798581 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.776333 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:21:27.799229 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.799201 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:21:27.804649 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.804622 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8t9n\" (UniqueName: \"kubernetes.io/projected/257eedd7-4d1b-4e1f-91c5-b5af649dc927-kube-api-access-f8t9n\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:21:27.807280 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.807233 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:21:27.853946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.840538 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:21:27.855086 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.855056 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:21:27.856563 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.856536 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:21:27.856748 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.856733 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:21:27.856848 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.856836 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:21:27.856953 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.845489 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:21:27.857054 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.853114 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 00:21:27.857208 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.857159 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:21:27.857270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.857249 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:21:27.857333 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.857309 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:21:27.857446 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.857415 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx946\" (UniqueName: \"kubernetes.io/projected/9acc881e-944d-4cdb-a67c-86c87d85d9e5-kube-api-access-fx946\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:21:27.905449 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:27.905404 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd"] Oct 30 00:21:28.063474 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.062207 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:21:28.064777 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.064469 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:21:28.097069 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.097028 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:21:28.175277 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.175245 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:21:28.175484 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.175463 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:21:28.175733 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.175715 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:21:28.176805 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.176781 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:21:28.177003 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.176983 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:21:28.177229 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.177211 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx946\" (UniqueName: \"kubernetes.io/projected/9acc881e-944d-4cdb-a67c-86c87d85d9e5-kube-api-access-fx946\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:21:28.177481 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.177464 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:21:28.199934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.182430 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:21:28.199934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.142397 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:21:28.199934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.182824 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:21:28.199934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.143156 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:21:28.199934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.183167 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:21:28.199934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.183612 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:21:28.262936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.260299 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx946\" (UniqueName: \"kubernetes.io/projected/9acc881e-944d-4cdb-a67c-86c87d85d9e5-kube-api-access-fx946\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:21:28.305609 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.305570 5001 patch_prober.go:29] interesting pod/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 00:21:28.305919 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.305865 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" podUID="f42670c1-7716-46d8-9bc3-3f06cf8caa8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 00:21:28.343044 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.309423 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:21:28.343044 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.309621 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6ad69a-w2bh7"] Oct 30 00:21:28.343044 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.314744 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" Oct 30 00:21:28.345558 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.345522 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:21:28.345856 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.345822 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:21:28.353458 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.353435 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:21:28.364941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.358198 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:21:28.364941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.359827 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:21:28.457159 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.454182 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:28.457723 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.457698 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:21:28.461746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.461429 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6ad69a-w2bh7"] Oct 30 00:21:28.476161 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.476091 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:21:28.500247 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.500001 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:21:28.505619 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.502633 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:28.652621 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.639468 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:21:28.664608 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.660307 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-493311b8-62hdl"] Oct 30 00:21:28.688789 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.671068 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-493311b8-62hdl"] Oct 30 00:21:28.688789 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.671193 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:21:28.947986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.947047 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a72a2dca-19ca-4a54-a217-936ef3a2e613" path="/var/lib/kubelet/pods/a72a2dca-19ca-4a54-a217-936ef3a2e613/volumes" Oct 30 00:21:28.947986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.947797 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9c8366d-ccbd-4047-8172-38a7a9d29909" path="/var/lib/kubelet/pods/e9c8366d-ccbd-4047-8172-38a7a9d29909/volumes" Oct 30 00:21:28.948448 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:21:28.948402 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257eedd7_4d1b_4e1f_91c5_b5af649dc927.slice/crio-2d309257816785c4dff62b3c8de05ef3d966a83e09d054b73f9fc86f39977e8b WatchSource:0}: Error finding container 2d309257816785c4dff62b3c8de05ef3d966a83e09d054b73f9fc86f39977e8b: Status 404 returned error can't find the container with id 2d309257816785c4dff62b3c8de05ef3d966a83e09d054b73f9fc86f39977e8b Oct 30 00:21:28.951774 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.951385 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:21:28.951774 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.951454 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:21:28.951774 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.951483 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:21:28.952273 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.952250 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s"] Oct 30 00:21:28.992593 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:21:28.992552 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9acc881e_944d_4cdb_a67c_86c87d85d9e5.slice/crio-c57f945bba2b27aab3e100310355f32502beea29a813b2ea53ffd1ab82da5743 WatchSource:0}: Error finding container c57f945bba2b27aab3e100310355f32502beea29a813b2ea53ffd1ab82da5743: Status 404 returned error can't find the container with id c57f945bba2b27aab3e100310355f32502beea29a813b2ea53ffd1ab82da5743 Oct 30 00:21:28.997036 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.996812 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:21:28.998042 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.996905 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:21:28.998042 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:28.997582 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:21:29.120284 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:29.119008 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd"] Oct 30 00:21:29.315965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:29.307623 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:21:29.315965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:29.308128 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:21:29.315965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:29.309036 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:21:29.786661 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:29.786618 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" event={"ID":"9acc881e-944d-4cdb-a67c-86c87d85d9e5","Type":"ContainerStarted","Data":"18e650f0ad0b3bf970b283925419150dec9a11eb5a4585b0f75b3b0b54756f74"} Oct 30 00:21:29.787199 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:29.786687 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" event={"ID":"9acc881e-944d-4cdb-a67c-86c87d85d9e5","Type":"ContainerStarted","Data":"c57f945bba2b27aab3e100310355f32502beea29a813b2ea53ffd1ab82da5743"} Oct 30 00:21:29.788526 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:29.788498 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" event={"ID":"257eedd7-4d1b-4e1f-91c5-b5af649dc927","Type":"ContainerStarted","Data":"b9e8adf6739248ea3335007c7091520c85e2c3957c53847b7b595612c8b0ab22"} Oct 30 00:21:29.788668 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:29.788541 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" event={"ID":"257eedd7-4d1b-4e1f-91c5-b5af649dc927","Type":"ContainerStarted","Data":"2d309257816785c4dff62b3c8de05ef3d966a83e09d054b73f9fc86f39977e8b"} Oct 30 00:21:29.886693 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:29.886621 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" podStartSLOduration=4.88660167 podStartE2EDuration="4.88660167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:29.844335609 +0000 UTC m=+1005.049836792" watchObservedRunningTime="2026-10-30 00:21:29.88660167 +0000 UTC m=+1005.092102854" Oct 30 00:21:29.890368 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:29.890322 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" podStartSLOduration=4.890308009 podStartE2EDuration="4.890308009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:29.883822749 +0000 UTC m=+1005.089323926" watchObservedRunningTime="2026-10-30 00:21:29.890308009 +0000 UTC m=+1005.095809189" Oct 30 00:21:29.996273 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:29.996243 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3fae6d-5xwgq"] Oct 30 00:21:30.001074 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:30.001054 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3fae6d-5xwgq" Oct 30 00:21:30.040248 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:30.040098 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:21:30.040248 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:30.040126 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:21:30.040248 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:30.040186 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:21:30.040248 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:30.040197 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:21:30.048063 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:30.048031 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3fae6d-5xwgq"] Oct 30 00:21:30.063103 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:30.062966 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:21:30.065350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:30.065146 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 00:21:30.065350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:30.065197 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" podUID="257eedd7-4d1b-4e1f-91c5-b5af649dc927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 00:21:30.102789 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:30.102715 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72fdac6e-341f-4f95-a281-9137884a57d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3fae6d-5xwgq\" (UID: \"72fdac6e-341f-4f95-a281-9137884a57d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3fae6d-5xwgq" Oct 30 00:21:30.103006 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:30.102980 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72fdac6e-341f-4f95-a281-9137884a57d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3fae6d-5xwgq\" (UID: \"72fdac6e-341f-4f95-a281-9137884a57d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3fae6d-5xwgq" Oct 30 00:21:30.103099 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:30.103085 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhkqf\" (UniqueName: \"kubernetes.io/projected/72fdac6e-341f-4f95-a281-9137884a57d8-kube-api-access-mhkqf\") pod \"maas-api-revoke-keys-e2e-api-a-3fae6d-5xwgq\" (UID: \"72fdac6e-341f-4f95-a281-9137884a57d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3fae6d-5xwgq" Oct 30 00:21:30.205599 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:30.204980 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72fdac6e-341f-4f95-a281-9137884a57d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3fae6d-5xwgq\" (UID: \"72fdac6e-341f-4f95-a281-9137884a57d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3fae6d-5xwgq" Oct 30 00:21:30.205599 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:30.205127 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhkqf\" (UniqueName: \"kubernetes.io/projected/72fdac6e-341f-4f95-a281-9137884a57d8-kube-api-access-mhkqf\") pod \"maas-api-revoke-keys-e2e-api-a-3fae6d-5xwgq\" (UID: \"72fdac6e-341f-4f95-a281-9137884a57d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3fae6d-5xwgq" Oct 30 00:21:30.205599 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:30.205255 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72fdac6e-341f-4f95-a281-9137884a57d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3fae6d-5xwgq\" (UID: \"72fdac6e-341f-4f95-a281-9137884a57d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3fae6d-5xwgq" Oct 30 00:21:30.206167 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:30.206137 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72fdac6e-341f-4f95-a281-9137884a57d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3fae6d-5xwgq\" (UID: \"72fdac6e-341f-4f95-a281-9137884a57d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3fae6d-5xwgq" Oct 30 00:21:30.229855 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:30.229821 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhkqf\" (UniqueName: \"kubernetes.io/projected/72fdac6e-341f-4f95-a281-9137884a57d8-kube-api-access-mhkqf\") pod \"maas-api-revoke-keys-e2e-api-a-3fae6d-5xwgq\" (UID: \"72fdac6e-341f-4f95-a281-9137884a57d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3fae6d-5xwgq" Oct 30 00:21:30.239185 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:30.230366 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72fdac6e-341f-4f95-a281-9137884a57d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3fae6d-5xwgq\" (UID: \"72fdac6e-341f-4f95-a281-9137884a57d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3fae6d-5xwgq" Oct 30 00:21:30.359652 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:30.359547 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3fae6d-5xwgq" Oct 30 00:21:30.503331 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:30.503299 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:21:30.504343 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:30.504314 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 00:21:30.504435 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:30.504375 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" podUID="9acc881e-944d-4cdb-a67c-86c87d85d9e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 00:21:31.091847 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:31.091810 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 00:21:31.213333 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:31.206111 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" podUID="257eedd7-4d1b-4e1f-91c5-b5af649dc927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 00:21:31.497165 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:31.497128 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3fae6d-5xwgq"] Oct 30 00:21:31.502958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:31.502930 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 00:21:31.503161 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:31.503136 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" podUID="9acc881e-944d-4cdb-a67c-86c87d85d9e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 00:21:32.068217 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:32.067104 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 00:21:32.068217 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:32.067167 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" podUID="257eedd7-4d1b-4e1f-91c5-b5af649dc927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 00:21:32.453671 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:32.445168 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3fae6d-5xwgq" event={"ID":"72fdac6e-341f-4f95-a281-9137884a57d8","Type":"ContainerStarted","Data":"b393c3711c124d85b3c1d35c1df3dc2141b77d8c1ed57574b4ead8663a3734e7"} Oct 30 00:21:32.456116 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:32.456055 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3fae6d-5xwgq" event={"ID":"72fdac6e-341f-4f95-a281-9137884a57d8","Type":"ContainerStarted","Data":"81142212d09c5c78eaf79af121d3d1ff820c15ebbbd47ca212fe91c9b3770d0f"} Oct 30 00:21:32.601727 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:32.558410 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 00:21:32.601727 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:32.558463 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" podUID="9acc881e-944d-4cdb-a67c-86c87d85d9e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 00:21:32.652935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:32.641835 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7afafb7b-xg76q"] Oct 30 00:21:32.672821 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:32.672759 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3fae6d-5xwgq" podStartSLOduration=3.672743932 podStartE2EDuration="3.672743932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:32.514376818 +0000 UTC m=+1007.719878000" watchObservedRunningTime="2026-10-30 00:21:32.672743932 +0000 UTC m=+1007.878245154" Oct 30 00:21:32.680414 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:32.680384 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7afafb7b-xg76q" Oct 30 00:21:32.688740 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:32.688706 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7afafb7b-xg76q"] Oct 30 00:21:32.765736 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:32.765416 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aae7ddca-8ecc-4d30-809a-23b2b39b506b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7afafb7b-xg76q\" (UID: \"aae7ddca-8ecc-4d30-809a-23b2b39b506b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7afafb7b-xg76q" Oct 30 00:21:32.765736 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:32.765511 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mc55\" (UniqueName: \"kubernetes.io/projected/aae7ddca-8ecc-4d30-809a-23b2b39b506b-kube-api-access-2mc55\") pod \"maas-api-revoke-sub-7afafb7b-xg76q\" (UID: \"aae7ddca-8ecc-4d30-809a-23b2b39b506b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7afafb7b-xg76q" Oct 30 00:21:32.765736 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:32.765707 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aae7ddca-8ecc-4d30-809a-23b2b39b506b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7afafb7b-xg76q\" (UID: \"aae7ddca-8ecc-4d30-809a-23b2b39b506b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7afafb7b-xg76q" Oct 30 00:21:32.867555 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:32.867523 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aae7ddca-8ecc-4d30-809a-23b2b39b506b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7afafb7b-xg76q\" (UID: \"aae7ddca-8ecc-4d30-809a-23b2b39b506b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7afafb7b-xg76q" Oct 30 00:21:32.867980 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:32.867953 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aae7ddca-8ecc-4d30-809a-23b2b39b506b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7afafb7b-xg76q\" (UID: \"aae7ddca-8ecc-4d30-809a-23b2b39b506b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7afafb7b-xg76q" Oct 30 00:21:32.868073 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:32.868047 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mc55\" (UniqueName: \"kubernetes.io/projected/aae7ddca-8ecc-4d30-809a-23b2b39b506b-kube-api-access-2mc55\") pod \"maas-api-revoke-sub-7afafb7b-xg76q\" (UID: \"aae7ddca-8ecc-4d30-809a-23b2b39b506b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7afafb7b-xg76q" Oct 30 00:21:32.868402 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:32.868357 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aae7ddca-8ecc-4d30-809a-23b2b39b506b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7afafb7b-xg76q\" (UID: \"aae7ddca-8ecc-4d30-809a-23b2b39b506b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7afafb7b-xg76q" Oct 30 00:21:32.870686 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:32.870656 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aae7ddca-8ecc-4d30-809a-23b2b39b506b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7afafb7b-xg76q\" (UID: \"aae7ddca-8ecc-4d30-809a-23b2b39b506b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7afafb7b-xg76q" Oct 30 00:21:32.889019 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:32.888994 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mc55\" (UniqueName: \"kubernetes.io/projected/aae7ddca-8ecc-4d30-809a-23b2b39b506b-kube-api-access-2mc55\") pod \"maas-api-revoke-sub-7afafb7b-xg76q\" (UID: \"aae7ddca-8ecc-4d30-809a-23b2b39b506b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7afafb7b-xg76q" Oct 30 00:21:32.999283 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:32.999249 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7afafb7b-xg76q" Oct 30 00:21:33.065620 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:33.064287 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 00:21:33.065620 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:33.064348 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" podUID="257eedd7-4d1b-4e1f-91c5-b5af649dc927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 00:21:33.465033 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:21:33.464992 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaae7ddca_8ecc_4d30_809a_23b2b39b506b.slice/crio-82ffa21ca67c50562a164bf5dc99066caf971d27628d08d463f42841dbbabfc9 WatchSource:0}: Error finding container 82ffa21ca67c50562a164bf5dc99066caf971d27628d08d463f42841dbbabfc9: Status 404 returned error can't find the container with id 82ffa21ca67c50562a164bf5dc99066caf971d27628d08d463f42841dbbabfc9 Oct 30 00:21:33.470181 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:33.470150 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7afafb7b-xg76q"] Oct 30 00:21:33.478009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:33.477984 5001 generic.go:356] "Generic (PLEG): container finished" podID="72fdac6e-341f-4f95-a281-9137884a57d8" containerID="b393c3711c124d85b3c1d35c1df3dc2141b77d8c1ed57574b4ead8663a3734e7" exitCode=0 Oct 30 00:21:33.478195 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:33.478169 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3fae6d-5xwgq" event={"ID":"72fdac6e-341f-4f95-a281-9137884a57d8","Type":"ContainerDied","Data":"b393c3711c124d85b3c1d35c1df3dc2141b77d8c1ed57574b4ead8663a3734e7"} Oct 30 00:21:33.503265 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:33.503234 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 00:21:33.503426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:33.503292 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" podUID="9acc881e-944d-4cdb-a67c-86c87d85d9e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 00:21:34.069964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:34.064039 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 00:21:34.069964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:34.064098 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" podUID="257eedd7-4d1b-4e1f-91c5-b5af649dc927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 00:21:34.539999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:34.531204 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 00:21:34.539999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:34.531269 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" podUID="9acc881e-944d-4cdb-a67c-86c87d85d9e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 00:21:34.591721 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:34.591687 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7afafb7b-xg76q" event={"ID":"aae7ddca-8ecc-4d30-809a-23b2b39b506b","Type":"ContainerStarted","Data":"6e92dd1e84a9c387c52b35687018c638b294542e77c8e7e7c8fb2fc7bbf852ec"} Oct 30 00:21:34.591956 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:34.591940 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7afafb7b-xg76q" event={"ID":"aae7ddca-8ecc-4d30-809a-23b2b39b506b","Type":"ContainerStarted","Data":"82ffa21ca67c50562a164bf5dc99066caf971d27628d08d463f42841dbbabfc9"} Oct 30 00:21:34.972820 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:34.972790 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3fae6d-5xwgq" Oct 30 00:21:34.995410 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:34.995379 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72fdac6e-341f-4f95-a281-9137884a57d8-kube-api-access-mhkqf\" (UniqueName: \"kubernetes.io/projected/72fdac6e-341f-4f95-a281-9137884a57d8-kube-api-access-mhkqf\") pod \"72fdac6e-341f-4f95-a281-9137884a57d8\" (UID: \"72fdac6e-341f-4f95-a281-9137884a57d8\") " Oct 30 00:21:34.995669 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:34.995650 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72fdac6e-341f-4f95-a281-9137884a57d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72fdac6e-341f-4f95-a281-9137884a57d8-maas-api-openshift-ca\") pod \"72fdac6e-341f-4f95-a281-9137884a57d8\" (UID: \"72fdac6e-341f-4f95-a281-9137884a57d8\") " Oct 30 00:21:34.995812 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:34.995795 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72fdac6e-341f-4f95-a281-9137884a57d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72fdac6e-341f-4f95-a281-9137884a57d8-maas-api-xks-ca\") pod \"72fdac6e-341f-4f95-a281-9137884a57d8\" (UID: \"72fdac6e-341f-4f95-a281-9137884a57d8\") " Oct 30 00:21:34.996057 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:34.996034 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72fdac6e-341f-4f95-a281-9137884a57d8-maas-api-openshift-ca" pod "72fdac6e-341f-4f95-a281-9137884a57d8" (UID: "72fdac6e-341f-4f95-a281-9137884a57d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:21:34.996516 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:34.996495 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72fdac6e-341f-4f95-a281-9137884a57d8-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:21:34.998036 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:34.998012 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72fdac6e-341f-4f95-a281-9137884a57d8-kube-api-access-mhkqf" pod "72fdac6e-341f-4f95-a281-9137884a57d8" (UID: "72fdac6e-341f-4f95-a281-9137884a57d8"). InnerVolumeSpecName "kube-api-access-mhkqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:21:34.998408 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:34.998390 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72fdac6e-341f-4f95-a281-9137884a57d8-maas-api-xks-ca" pod "72fdac6e-341f-4f95-a281-9137884a57d8" (UID: "72fdac6e-341f-4f95-a281-9137884a57d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:21:35.004080 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:35.004014 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7afafb7b-xg76q" podStartSLOduration=3.003999441 podStartE2EDuration="3.003999441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:34.669203297 +0000 UTC m=+1009.874704481" watchObservedRunningTime="2026-10-30 00:21:35.003999441 +0000 UTC m=+1010.209500625" Oct 30 00:21:35.078086 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:35.078043 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 00:21:35.078291 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:35.078131 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" podUID="257eedd7-4d1b-4e1f-91c5-b5af649dc927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 00:21:35.097826 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:35.097746 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72fdac6e-341f-4f95-a281-9137884a57d8-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:21:35.097826 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:35.097779 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhkqf\" (UniqueName: \"kubernetes.io/projected/72fdac6e-341f-4f95-a281-9137884a57d8-kube-api-access-mhkqf\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:21:35.510991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:35.510301 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 00:21:35.510991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:35.510368 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" podUID="9acc881e-944d-4cdb-a67c-86c87d85d9e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 00:21:35.684567 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:35.684517 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3fae6d-5xwgq" event={"ID":"72fdac6e-341f-4f95-a281-9137884a57d8","Type":"ContainerDied","Data":"81142212d09c5c78eaf79af121d3d1ff820c15ebbbd47ca212fe91c9b3770d0f"} Oct 30 00:21:35.685095 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:35.684591 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81142212d09c5c78eaf79af121d3d1ff820c15ebbbd47ca212fe91c9b3770d0f" Oct 30 00:21:35.685095 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:35.684668 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3fae6d-5xwgq" Oct 30 00:21:35.702646 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:35.702611 5001 generic.go:356] "Generic (PLEG): container finished" podID="aae7ddca-8ecc-4d30-809a-23b2b39b506b" containerID="6e92dd1e84a9c387c52b35687018c638b294542e77c8e7e7c8fb2fc7bbf852ec" exitCode=0 Oct 30 00:21:35.703379 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:35.703051 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7afafb7b-xg76q" event={"ID":"aae7ddca-8ecc-4d30-809a-23b2b39b506b","Type":"ContainerDied","Data":"6e92dd1e84a9c387c52b35687018c638b294542e77c8e7e7c8fb2fc7bbf852ec"} Oct 30 00:21:36.063572 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:36.063437 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 00:21:36.063740 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:36.063564 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" podUID="257eedd7-4d1b-4e1f-91c5-b5af649dc927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 00:21:36.502545 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:36.502445 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 00:21:36.502545 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:36.502528 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" podUID="9acc881e-944d-4cdb-a67c-86c87d85d9e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 00:21:37.080861 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.080817 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 00:21:37.101850 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.092987 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" podUID="257eedd7-4d1b-4e1f-91c5-b5af649dc927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 00:21:37.315502 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.315463 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g"] Oct 30 00:21:37.316497 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.316463 5001 cpu_manager.go:414] "containerMap: removing container" podUID="72fdac6e-341f-4f95-a281-9137884a57d8" containerName="revoke-keys" Oct 30 00:21:37.316606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.316596 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72fdac6e-341f-4f95-a281-9137884a57d8" containerName="revoke-keys" Oct 30 00:21:37.316812 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.316803 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="72fdac6e-341f-4f95-a281-9137884a57d8" containerName="revoke-keys" Oct 30 00:21:37.323833 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.323087 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:21:37.345762 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.343813 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-0aea47-openshift-default-dockercfg-2z5cc\"" Oct 30 00:21:37.355903 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.354228 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g"] Oct 30 00:21:37.628232 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.615550 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 00:21:37.628232 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.615606 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" podUID="9acc881e-944d-4cdb-a67c-86c87d85d9e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 00:21:37.628232 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.620766 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istio-podinfo\") pod \"e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:21:37.628232 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.620812 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istio-envoy\") pod \"e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:21:37.628232 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.620947 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbxmg\" (UniqueName: \"kubernetes.io/projected/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-kube-api-access-rbxmg\") pod \"e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:21:37.628232 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.621078 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-workload-certs\") pod \"e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:21:37.628232 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.621214 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-credential-socket\") pod \"e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:21:37.628232 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.621285 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istio-data\") pod \"e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:21:37.628232 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.621361 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-workload-socket\") pod \"e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:21:37.628232 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.621421 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istio-token\") pod \"e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:21:37.628232 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.621456 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istiod-ca-cert\") pod \"e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:21:37.738632 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.731063 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbxmg\" (UniqueName: \"kubernetes.io/projected/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-kube-api-access-rbxmg\") pod \"e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:21:37.738632 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.731143 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-workload-certs\") pod \"e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:21:37.738632 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.731277 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-credential-socket\") pod \"e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:21:37.738632 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.731348 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istio-data\") pod \"e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:21:37.738632 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.731488 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-workload-socket\") pod \"e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:21:37.738632 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.731617 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istio-token\") pod \"e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:21:37.738632 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.731658 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istiod-ca-cert\") pod \"e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:21:37.738632 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.731766 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istio-podinfo\") pod \"e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:21:37.738632 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.731793 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istio-envoy\") pod \"e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:21:37.738632 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.732849 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istio-data\") pod \"e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:21:37.738632 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.733449 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-workload-certs\") pod \"e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:21:37.738632 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.733690 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-credential-socket\") pod \"e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:21:37.788699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.769276 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-workload-socket\") pod \"e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:21:37.788699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.769906 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istiod-ca-cert\") pod \"e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:21:37.829935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.826478 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istio-podinfo\") pod \"e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:21:37.889556 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.867467 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istio-token\") pod \"e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:21:37.895103 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.868692 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istio-envoy\") pod \"e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:21:37.895335 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.872182 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbxmg\" (UniqueName: \"kubernetes.io/projected/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-kube-api-access-rbxmg\") pod \"e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:21:37.949293 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:37.948892 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7afafb7b-xg76q" Oct 30 00:21:38.006984 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:38.000841 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:21:38.006984 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:38.001262 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7afafb7b-xg76q" Oct 30 00:21:38.006984 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:38.001430 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7afafb7b-xg76q" event={"ID":"aae7ddca-8ecc-4d30-809a-23b2b39b506b","Type":"ContainerDied","Data":"82ffa21ca67c50562a164bf5dc99066caf971d27628d08d463f42841dbbabfc9"} Oct 30 00:21:38.006984 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:38.001472 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82ffa21ca67c50562a164bf5dc99066caf971d27628d08d463f42841dbbabfc9" Oct 30 00:21:38.056746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:38.055498 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eebf0d77-h7c9s"] Oct 30 00:21:38.065895 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:38.065472 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 00:21:38.065895 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:38.065529 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" podUID="257eedd7-4d1b-4e1f-91c5-b5af649dc927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 00:21:38.142563 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:38.142454 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eebf0d77-h7c9s"] Oct 30 00:21:38.324635 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:38.324597 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aae7ddca-8ecc-4d30-809a-23b2b39b506b-kube-api-access-2mc55\" (UniqueName: \"kubernetes.io/projected/aae7ddca-8ecc-4d30-809a-23b2b39b506b-kube-api-access-2mc55\") pod \"aae7ddca-8ecc-4d30-809a-23b2b39b506b\" (UID: \"aae7ddca-8ecc-4d30-809a-23b2b39b506b\") " Oct 30 00:21:38.421218 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:38.382079 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aae7ddca-8ecc-4d30-809a-23b2b39b506b-kube-api-access-2mc55" pod "aae7ddca-8ecc-4d30-809a-23b2b39b506b" (UID: "aae7ddca-8ecc-4d30-809a-23b2b39b506b"). InnerVolumeSpecName "kube-api-access-2mc55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:21:38.421369 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:38.421269 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aae7ddca-8ecc-4d30-809a-23b2b39b506b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aae7ddca-8ecc-4d30-809a-23b2b39b506b-maas-api-xks-ca\") pod \"aae7ddca-8ecc-4d30-809a-23b2b39b506b\" (UID: \"aae7ddca-8ecc-4d30-809a-23b2b39b506b\") " Oct 30 00:21:38.421674 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:38.421651 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aae7ddca-8ecc-4d30-809a-23b2b39b506b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aae7ddca-8ecc-4d30-809a-23b2b39b506b-maas-api-openshift-ca\") pod \"aae7ddca-8ecc-4d30-809a-23b2b39b506b\" (UID: \"aae7ddca-8ecc-4d30-809a-23b2b39b506b\") " Oct 30 00:21:38.434599 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:38.433773 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mc55\" (UniqueName: \"kubernetes.io/projected/aae7ddca-8ecc-4d30-809a-23b2b39b506b-kube-api-access-2mc55\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:21:38.434599 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:38.422182 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aae7ddca-8ecc-4d30-809a-23b2b39b506b-maas-api-openshift-ca" pod "aae7ddca-8ecc-4d30-809a-23b2b39b506b" (UID: "aae7ddca-8ecc-4d30-809a-23b2b39b506b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:21:38.434599 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:38.431779 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aae7ddca-8ecc-4d30-809a-23b2b39b506b-maas-api-xks-ca" pod "aae7ddca-8ecc-4d30-809a-23b2b39b506b" (UID: "aae7ddca-8ecc-4d30-809a-23b2b39b506b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:21:38.563239 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:38.562953 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aae7ddca-8ecc-4d30-809a-23b2b39b506b-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:21:38.563495 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:38.563474 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aae7ddca-8ecc-4d30-809a-23b2b39b506b-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:21:38.691973 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:38.691797 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g"] Oct 30 00:21:38.698622 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:38.698580 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:21:38.698808 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:38.698788 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:21:38.698966 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:38.698948 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:21:38.983980 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:38.983112 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f91c3a1-351a-4939-86c8-f6864f38b6b3" path="/var/lib/kubelet/pods/7f91c3a1-351a-4939-86c8-f6864f38b6b3/volumes" Oct 30 00:21:39.220379 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:39.220338 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" event={"ID":"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7","Type":"ContainerStarted","Data":"d8349c31038fd354f846c7f57f925a6d777d8112a1b5b84ce9d9c75c0a5f42ad"} Oct 30 00:21:39.221034 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:39.221004 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" event={"ID":"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7","Type":"ContainerStarted","Data":"194adfb9bd691a607cfd09475c3511d9cc32fef82d73cc9c1fca6aa7e4741aef"} Oct 30 00:21:39.274212 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:39.274148 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" podStartSLOduration=2.2741275930000002 podStartE2EDuration="2.274127593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:21:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:39.261521153 +0000 UTC m=+1014.467022360" watchObservedRunningTime="2026-10-30 00:21:39.274127593 +0000 UTC m=+1014.479628778" Oct 30 00:21:39.381936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:39.381308 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:21:39.389946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:39.383035 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:21:39.408002 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:39.398442 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:21:39.814703 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:39.814509 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:21:39.815119 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:39.815099 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:21:39.816179 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:39.816158 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:21:40.002084 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:40.002052 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:21:40.008604 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:40.008573 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:21:40.234294 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:40.234207 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:21:40.235449 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:40.235419 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:21:41.666136 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:41.666093 5001 patch_prober.go:29] interesting pod/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:21:41.666684 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:41.666168 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" podUID="97f84a41-7ac2-4734-ab36-ebf29a62b8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:42.495336 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:42.495296 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk"] Oct 30 00:21:42.497537 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:42.496566 5001 cpu_manager.go:414] "containerMap: removing container" podUID="aae7ddca-8ecc-4d30-809a-23b2b39b506b" containerName="revoke-keys" Oct 30 00:21:42.497537 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:42.496588 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aae7ddca-8ecc-4d30-809a-23b2b39b506b" containerName="revoke-keys" Oct 30 00:21:42.497537 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:42.496766 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="aae7ddca-8ecc-4d30-809a-23b2b39b506b" containerName="revoke-keys" Oct 30 00:21:42.503767 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:42.502189 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:21:42.532937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:42.527631 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk"] Oct 30 00:21:42.553620 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:42.553585 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-workload-certs\") pod \"e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:21:42.553775 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:42.553649 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jp7x\" (UniqueName: \"kubernetes.io/projected/03f8426e-b14c-43f9-9598-04b5b7b309b5-kube-api-access-9jp7x\") pod \"e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:21:42.553775 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:42.553740 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03f8426e-b14c-43f9-9598-04b5b7b309b5-istio-token\") pod \"e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:21:42.553925 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:42.553803 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-workload-socket\") pod \"e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:21:42.553925 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:42.553863 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-credential-socket\") pod \"e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:21:42.554061 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:42.553979 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-istio-data\") pod \"e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:21:42.554061 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:42.554051 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-istio-envoy\") pod \"e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:21:42.554171 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:42.554143 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f8426e-b14c-43f9-9598-04b5b7b309b5-istiod-ca-cert\") pod \"e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:21:42.554228 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:42.554174 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f8426e-b14c-43f9-9598-04b5b7b309b5-istio-podinfo\") pod \"e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:21:42.578301 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:42.578271 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-08dd33-openshift-default-dockercfg-kfg57\"" Oct 30 00:21:42.883930 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:42.866755 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-istio-data\") pod \"e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:21:42.883930 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:42.866836 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-istio-envoy\") pod \"e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:21:42.883930 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:42.872969 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f8426e-b14c-43f9-9598-04b5b7b309b5-istiod-ca-cert\") pod \"e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:21:42.883930 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:42.873033 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f8426e-b14c-43f9-9598-04b5b7b309b5-istio-podinfo\") pod \"e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:21:42.883930 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:42.873170 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-workload-certs\") pod \"e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:21:42.883930 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:42.873217 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jp7x\" (UniqueName: \"kubernetes.io/projected/03f8426e-b14c-43f9-9598-04b5b7b309b5-kube-api-access-9jp7x\") pod \"e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:21:42.883930 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:42.873320 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03f8426e-b14c-43f9-9598-04b5b7b309b5-istio-token\") pod \"e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:21:42.883930 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:42.873391 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-workload-socket\") pod \"e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:21:42.883930 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:42.873456 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-credential-socket\") pod \"e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:21:42.907243 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:42.898728 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-istio-data\") pod \"e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:21:42.917389 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:42.916851 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f8426e-b14c-43f9-9598-04b5b7b309b5-istiod-ca-cert\") pod \"e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:21:42.931939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:42.922573 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-credential-socket\") pod \"e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:21:43.005140 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:42.941697 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-workload-socket\") pod \"e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:21:43.005140 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:42.962943 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-workload-certs\") pod \"e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:21:43.005140 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:42.971975 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03f8426e-b14c-43f9-9598-04b5b7b309b5-istio-token\") pod \"e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:21:43.005140 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:42.972589 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jp7x\" (UniqueName: \"kubernetes.io/projected/03f8426e-b14c-43f9-9598-04b5b7b309b5-kube-api-access-9jp7x\") pod \"e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:21:43.040851 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:43.037451 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-istio-envoy\") pod \"e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:21:43.059032 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:43.055592 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f8426e-b14c-43f9-9598-04b5b7b309b5-istio-podinfo\") pod \"e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:21:43.151986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:43.150826 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:21:43.451995 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:43.450964 5001 generic.go:356] "Generic (PLEG): container finished" podID="97f84a41-7ac2-4734-ab36-ebf29a62b8fa" containerID="4068448d5f41d3e76598637ce899a36296453078dbfb02141c2fbcd30862b98d" exitCode=0 Oct 30 00:21:43.451995 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:43.451277 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" event={"ID":"97f84a41-7ac2-4734-ab36-ebf29a62b8fa","Type":"ContainerDied","Data":"4068448d5f41d3e76598637ce899a36296453078dbfb02141c2fbcd30862b98d"} Oct 30 00:21:43.451995 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:43.451844 5001 scope.go:122] "RemoveContainer" containerID="4068448d5f41d3e76598637ce899a36296453078dbfb02141c2fbcd30862b98d" Oct 30 00:21:43.486193 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:43.478564 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:21:43.486193 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:43.478705 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:21:43.486193 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:43.478757 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:21:44.381267 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.363992 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:44.444947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.444893 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4"] Oct 30 00:21:44.463800 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.463762 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:21:44.481715 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.481674 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f52ea1-openshift-default-dockercfg-6zrs8\"" Oct 30 00:21:44.494465 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.494388 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:21:44.494718 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.494693 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:21:44.494856 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.494839 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:21:44.495698 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.495680 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4"] Oct 30 00:21:44.503512 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.503484 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" event={"ID":"03f8426e-b14c-43f9-9598-04b5b7b309b5","Type":"ContainerStarted","Data":"4f195e056c50eee7bb811b0c5fdc2aec666594d56c673b5556ee7a4c9bed2bf4"} Oct 30 00:21:44.547029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.529792 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk"] Oct 30 00:21:44.580935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.570185 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istio-envoy\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:21:44.580935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.570310 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-workload-socket\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:21:44.580935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.570352 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istio-data\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:21:44.580935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.570389 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istio-token\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:21:44.580935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.570426 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-credential-socket\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:21:44.580935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.570565 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd82c\" (UniqueName: \"kubernetes.io/projected/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-kube-api-access-fd82c\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:21:44.580935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.570623 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-workload-certs\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:21:44.580935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.570851 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istio-podinfo\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:21:44.580935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.570927 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istiod-ca-cert\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:21:44.708934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.708878 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istio-envoy\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:21:44.709353 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.709331 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-workload-socket\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:21:44.709505 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.709490 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istio-data\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:21:44.709628 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.709615 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istio-token\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:21:44.709820 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.709756 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-credential-socket\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:21:44.710114 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.710092 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd82c\" (UniqueName: \"kubernetes.io/projected/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-kube-api-access-fd82c\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:21:44.710467 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.710446 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-workload-certs\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:21:44.710754 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.710737 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istio-podinfo\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:21:44.710929 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.710897 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istiod-ca-cert\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:21:44.713307 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.713285 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istiod-ca-cert\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:21:44.714109 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.714090 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-credential-socket\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:21:44.714659 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.714642 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-workload-socket\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:21:44.715289 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.715270 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-workload-certs\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:21:44.716973 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.716954 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istio-data\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:21:44.735535 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.735496 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istio-podinfo\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:21:44.753970 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.753936 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd82c\" (UniqueName: \"kubernetes.io/projected/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-kube-api-access-fd82c\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:21:44.763149 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.763083 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istio-envoy\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:21:44.765947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.765904 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istio-token\") pod \"e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:21:44.859153 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:44.858773 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:21:45.275024 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:45.274989 5001 scope.go:122] "RemoveContainer" containerID="2c11bc80ded7787fbf161f3658b1dae8a1ccf78549c587486e82a847fc52b9c6" Oct 30 00:21:45.290323 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:45.290282 5001 scope.go:122] "RemoveContainer" containerID="24814a0b004d54d3cd15cfbf2dae0a6eaee087c5c1ee33b2dc6e2a1b6ae53ef7" Oct 30 00:21:45.304337 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:45.304242 5001 scope.go:122] "RemoveContainer" containerID="5f55214eb6a394e36c0e0361b03d0af1641c77c3efafd82cbff5d94050765e6a" Oct 30 00:21:45.318432 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:45.318405 5001 scope.go:122] "RemoveContainer" containerID="6dad701f3db32f43ee7cf1046254c21c4bed9c8a4d68c1bb5461ff809db8ddbb" Oct 30 00:21:45.384459 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:45.384427 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4"] Oct 30 00:21:45.395930 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:45.394120 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:21:45.395930 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:45.394215 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:21:45.395930 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:45.394265 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:21:45.515423 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:45.515388 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" event={"ID":"97f84a41-7ac2-4734-ab36-ebf29a62b8fa","Type":"ContainerStarted","Data":"d3be5d676efea82110af81195d4e7082d4262d35d85361fa99390aabeb519fbc"} Oct 30 00:21:45.520974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:45.519670 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" event={"ID":"ed3d03e4-5550-40bc-8c00-48c3c7927fa1","Type":"ContainerStarted","Data":"dc5d7805cd01f09ae7d448989b3fcd91eddda4a0896f21dd546d223c3494e577"} Oct 30 00:21:45.520974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:45.519730 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" event={"ID":"ed3d03e4-5550-40bc-8c00-48c3c7927fa1","Type":"ContainerStarted","Data":"e0451f46023e3d7f8c60ec34a16d61c4b8a86b1035ea19c91694d8a51eef89e2"} Oct 30 00:21:45.524957 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:45.522634 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" event={"ID":"03f8426e-b14c-43f9-9598-04b5b7b309b5","Type":"ContainerStarted","Data":"13b59302f73b01109636fec53bf8d139d4c2af5430bf221ebba254a665d0294c"} Oct 30 00:21:45.602811 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:45.601809 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" podStartSLOduration=3.601794687 podStartE2EDuration="3.601794687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:21:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:45.594727141 +0000 UTC m=+1020.800228348" watchObservedRunningTime="2026-10-30 00:21:45.601794687 +0000 UTC m=+1020.807295869" Oct 30 00:21:45.638855 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:45.638739 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" podStartSLOduration=2.638719224 podStartE2EDuration="2.638719224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:45.632928222 +0000 UTC m=+1020.838429404" watchObservedRunningTime="2026-10-30 00:21:45.638719224 +0000 UTC m=+1020.844220406" Oct 30 00:21:45.865935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:45.865405 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:21:46.289936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:46.284300 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:21:46.313292 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:46.295321 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:21:46.427934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:46.424987 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:21:46.427934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:46.425219 5001 patch_prober.go:29] interesting pod/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 00:21:46.427934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:46.425279 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" podUID="97f84a41-7ac2-4734-ab36-ebf29a62b8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 00:21:46.772934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:46.768118 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:21:46.772934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:46.770308 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:21:46.884799 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:46.883266 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:21:47.350932 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:21:47.350875 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:21:47.408055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:47.408014 5001 patch_prober.go:29] interesting pod/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 00:21:47.408252 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:47.408078 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" podUID="97f84a41-7ac2-4734-ab36-ebf29a62b8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 00:21:47.458800 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:47.458752 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9"] Oct 30 00:21:47.464654 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:47.464629 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:21:47.498466 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:47.492944 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c34304-openshift-default-dockercfg-v98n2\"" Oct 30 00:21:47.498466 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:47.495527 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9"] Oct 30 00:21:47.593575 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:47.590660 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-credential-socket\") pod \"e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:21:47.593575 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:47.590803 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh2xd\" (UniqueName: \"kubernetes.io/projected/4bfbc166-3dbf-4aae-96f5-515f59917846-kube-api-access-kh2xd\") pod \"e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:21:47.593575 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:47.590838 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bfbc166-3dbf-4aae-96f5-515f59917846-istiod-ca-cert\") pod \"e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:21:47.593575 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:47.590888 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-istio-envoy\") pod \"e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:21:47.593575 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:47.590974 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-istio-data\") pod \"e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:21:47.593575 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:47.591077 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-workload-certs\") pod \"e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:21:47.593575 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:47.591218 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bfbc166-3dbf-4aae-96f5-515f59917846-istio-token\") pod \"e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:21:47.593575 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:47.591345 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-workload-socket\") pod \"e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:21:47.593575 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:47.591419 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bfbc166-3dbf-4aae-96f5-515f59917846-istio-podinfo\") pod \"e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:21:47.703986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:47.703421 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-workload-socket\") pod \"e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:21:47.703986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:47.703506 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bfbc166-3dbf-4aae-96f5-515f59917846-istio-podinfo\") pod \"e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:21:47.703986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:47.703592 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-credential-socket\") pod \"e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:21:47.703986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:47.703703 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh2xd\" (UniqueName: \"kubernetes.io/projected/4bfbc166-3dbf-4aae-96f5-515f59917846-kube-api-access-kh2xd\") pod \"e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:21:47.703986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:47.703735 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bfbc166-3dbf-4aae-96f5-515f59917846-istiod-ca-cert\") pod \"e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:21:47.703986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:47.703784 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-istio-envoy\") pod \"e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:21:47.703986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:47.703851 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-istio-data\") pod \"e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:21:47.703986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:47.703886 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-workload-certs\") pod \"e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:21:47.704583 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:47.704011 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bfbc166-3dbf-4aae-96f5-515f59917846-istio-token\") pod \"e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:21:47.742059 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:47.728542 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-workload-socket\") pod \"e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:21:47.742059 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:47.730732 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-credential-socket\") pod \"e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:21:47.742059 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:47.734249 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-istio-data\") pod \"e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:21:47.742059 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:47.734523 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-workload-certs\") pod \"e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:21:47.763104 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:47.759389 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bfbc166-3dbf-4aae-96f5-515f59917846-istiod-ca-cert\") pod \"e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:21:48.083994 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:48.080516 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bfbc166-3dbf-4aae-96f5-515f59917846-istio-token\") pod \"e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:21:48.085198 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:48.085169 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bfbc166-3dbf-4aae-96f5-515f59917846-istio-podinfo\") pod \"e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:21:48.085598 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:48.085578 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-istio-envoy\") pod \"e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:21:48.152332 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:48.104599 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh2xd\" (UniqueName: \"kubernetes.io/projected/4bfbc166-3dbf-4aae-96f5-515f59917846-kube-api-access-kh2xd\") pod \"e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:21:48.212216 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:48.149270 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:21:48.212537 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:48.212517 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:21:48.453984 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:48.431099 5001 patch_prober.go:29] interesting pod/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 00:21:48.453984 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:48.431157 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" podUID="97f84a41-7ac2-4734-ab36-ebf29a62b8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 00:21:48.453984 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:48.431836 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:21:48.604831 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:48.604775 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:48.928154 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:48.928111 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:49.407892 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:49.407860 5001 patch_prober.go:29] interesting pod/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 00:21:49.408109 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:49.407947 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" podUID="97f84a41-7ac2-4734-ab36-ebf29a62b8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 00:21:49.468530 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:21:49.468496 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bfbc166_3dbf_4aae_96f5_515f59917846.slice/crio-50149fb4715117fa618e2d60cce683f67959d04e7b3f2b68092af06fe4e38a43 WatchSource:0}: Error finding container 50149fb4715117fa618e2d60cce683f67959d04e7b3f2b68092af06fe4e38a43: Status 404 returned error can't find the container with id 50149fb4715117fa618e2d60cce683f67959d04e7b3f2b68092af06fe4e38a43 Oct 30 00:21:49.470091 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:49.470067 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9"] Oct 30 00:21:49.471875 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:49.471839 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:21:49.472000 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:49.471920 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:21:49.472000 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:49.471960 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:21:50.469937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:50.456325 5001 patch_prober.go:29] interesting pod/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 00:21:50.469937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:50.456426 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" podUID="97f84a41-7ac2-4734-ab36-ebf29a62b8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 00:21:50.596670 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:50.587352 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" event={"ID":"4bfbc166-3dbf-4aae-96f5-515f59917846","Type":"ContainerStarted","Data":"af07da8cc8da63de4de9af460ea429fc7f10f344885d5ee21a8d9d1a291b8d3d"} Oct 30 00:21:50.596670 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:50.587429 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" event={"ID":"4bfbc166-3dbf-4aae-96f5-515f59917846","Type":"ContainerStarted","Data":"50149fb4715117fa618e2d60cce683f67959d04e7b3f2b68092af06fe4e38a43"} Oct 30 00:21:50.739818 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:50.739695 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" podStartSLOduration=3.73967313 podStartE2EDuration="3.73967313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:21:50.683054597 +0000 UTC m=+1025.888555794" watchObservedRunningTime="2026-10-30 00:21:50.73967313 +0000 UTC m=+1025.945174313" Oct 30 00:21:51.488895 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:51.488358 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:21:51.488895 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:51.488473 5001 patch_prober.go:29] interesting pod/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 00:21:51.488895 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:51.488513 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" podUID="97f84a41-7ac2-4734-ab36-ebf29a62b8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 00:21:51.546950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:51.546881 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:21:51.972963 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:51.970771 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:21:52.071936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:52.057156 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:21:52.423021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:52.414801 5001 patch_prober.go:29] interesting pod/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 00:21:52.423021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:52.414860 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" podUID="97f84a41-7ac2-4734-ab36-ebf29a62b8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 00:21:53.576394 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:53.576362 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:21:53.578482 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:53.578310 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:21:53.594223 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:53.594147 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:21:59.376712 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:59.376482 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:21:59.531076 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:59.531030 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 00:21:59.826990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:21:59.819163 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:00.339066 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:00.339018 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:00.381938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:00.378201 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:00.381938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:00.378799 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:22:00.403947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:00.403047 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:00.403947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:00.403459 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:00.527192 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:00.526214 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:00.532063 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:00.532028 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:00.540426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:00.532821 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:22:00.783108 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:00.771167 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" podUID="b3e66631-9562-4bb5-b005-1fc179a8582d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.135:8000/ready\": context deadline exceeded" Oct 30 00:22:00.836977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:00.832471 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:00.836977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:00.833209 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:22:01.351993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:01.345540 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:01.352904 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:01.352870 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:22:01.426846 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:01.426798 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:22:01.554934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:01.550983 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 00:22:01.561047 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:01.561016 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:22:01.609989 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:01.596627 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:22:01.611305 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:01.599003 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:01.623969 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:01.612005 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:22:01.626624 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:01.599053 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 30 00:22:01.640573 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:01.640544 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:22:01.660578 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:01.660538 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:01.750957 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:01.739421 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:22:01.788957 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:01.775404 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" podUID="b3e66631-9562-4bb5-b005-1fc179a8582d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.135:8000/ready\": context deadline exceeded" Oct 30 00:22:01.788957 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:01.776379 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" Oct 30 00:22:01.788957 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:01.788007 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:01.907972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:01.907832 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:02.368088 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:02.368038 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:22:02.564096 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:02.556156 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:02.616098 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:02.614989 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:02.617021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:02.616984 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:02.644764 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:02.644665 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:02.674077 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:02.674022 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:02.698095 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:02.698062 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:22:02.729025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:02.728993 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:22:02.776665 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:02.776621 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" podUID="b3e66631-9562-4bb5-b005-1fc179a8582d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.135:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:02.918137 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:02.917998 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:03.687084 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:03.667879 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:22:03.692522 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:03.692474 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:03.783938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:03.761471 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:22:03.811128 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:03.811075 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:03.831944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:03.822060 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:22:03.855975 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:03.847930 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:22:03.888948 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:03.874672 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" podUID="b3e66631-9562-4bb5-b005-1fc179a8582d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.135:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:03.987093 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:03.986978 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:04.322278 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:04.322224 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" podUID="b3e66631-9562-4bb5-b005-1fc179a8582d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.135:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:04.604498 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:04.604418 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" Oct 30 00:22:04.814932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:04.796778 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:04.814932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:04.797958 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:04.814932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:04.798273 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:04.840938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:04.817658 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:04.946985 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:04.913044 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:05.182602 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:05.180129 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:22:05.467936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:05.461071 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:05.664047 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:05.663999 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:05.803671 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:05.803616 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 00:22:05.821934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:05.805116 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:22:05.821934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:05.814095 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:22:05.940952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:05.940004 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:05.974943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:05.974030 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:06.006064 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:06.006012 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:06.138464 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:06.138376 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:22:06.182244 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:06.168965 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:22:06.187048 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:06.186463 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:22:06.195946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:06.187295 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:22:06.244933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:06.220947 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:06.244933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:06.221325 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:22:06.374283 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:06.374252 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:22:06.396991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:06.396891 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:22:06.402536 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:06.402504 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:22:07.597293 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:07.597227 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 00:22:08.386942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:08.374175 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:08.418002 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:08.417949 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:08.502095 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:08.502047 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:08.517967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:08.512014 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:08.714989 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:08.708322 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:08.825443 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:08.825395 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" podUID="b3e66631-9562-4bb5-b005-1fc179a8582d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.135:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:08.994081 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:08.993994 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7587-c250f6"] Oct 30 00:22:09.000647 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:09.000617 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7587-c250f6" Oct 30 00:22:09.005045 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:09.004869 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7587-c250f6"] Oct 30 00:22:09.014713 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:09.014688 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hr9mw\"" Oct 30 00:22:09.101753 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:09.101714 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nps7\" (UniqueName: \"kubernetes.io/projected/67e1393f-a46e-4289-9715-04d738c00a7e-kube-api-access-9nps7\") pod \"test-curl-7587-c250f6\" (UID: \"67e1393f-a46e-4289-9715-04d738c00a7e\") " pod="opendatahub/test-curl-7587-c250f6" Oct 30 00:22:09.203112 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:09.203080 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nps7\" (UniqueName: \"kubernetes.io/projected/67e1393f-a46e-4289-9715-04d738c00a7e-kube-api-access-9nps7\") pod \"test-curl-7587-c250f6\" (UID: \"67e1393f-a46e-4289-9715-04d738c00a7e\") " pod="opendatahub/test-curl-7587-c250f6" Oct 30 00:22:09.230538 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:09.230504 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nps7\" (UniqueName: \"kubernetes.io/projected/67e1393f-a46e-4289-9715-04d738c00a7e-kube-api-access-9nps7\") pod \"test-curl-7587-c250f6\" (UID: \"67e1393f-a46e-4289-9715-04d738c00a7e\") " pod="opendatahub/test-curl-7587-c250f6" Oct 30 00:22:09.327673 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:09.327636 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7587-c250f6" Oct 30 00:22:09.756437 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:22:09.756403 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67e1393f_a46e_4289_9715_04d738c00a7e.slice/crio-2742144d9b3f099b1c7535836ade5a97ab9b6843e0dfc85b6675ebd38cb8bfdf WatchSource:0}: Error finding container 2742144d9b3f099b1c7535836ade5a97ab9b6843e0dfc85b6675ebd38cb8bfdf: Status 404 returned error can't find the container with id 2742144d9b3f099b1c7535836ade5a97ab9b6843e0dfc85b6675ebd38cb8bfdf Oct 30 00:22:09.756877 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:09.756640 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7587-c250f6"] Oct 30 00:22:09.757163 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:09.756986 5001 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:22:10.723118 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:10.723076 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7587-c250f6" event={"ID":"67e1393f-a46e-4289-9715-04d738c00a7e","Type":"ContainerStarted","Data":"2742144d9b3f099b1c7535836ade5a97ab9b6843e0dfc85b6675ebd38cb8bfdf"} Oct 30 00:22:14.859797 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:14.859057 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7587-c250f6" event={"ID":"67e1393f-a46e-4289-9715-04d738c00a7e","Type":"ContainerStarted","Data":"d71b5f2e087e9225a1bb75d197324d03023fb1b8e7b1f4b447ef8114d3bf70d2"} Oct 30 00:22:14.895925 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:14.895865 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7587-c250f6" podStartSLOduration=5.009242849 podStartE2EDuration="8.89584782s" totalImagesPullingTime="3.886604971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:22:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:22:14.891316561 +0000 UTC m=+1050.096817743" watchObservedRunningTime="2026-10-30 00:22:14.89584782 +0000 UTC m=+1050.101349003" Oct 30 00:22:15.044230 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:15.044179 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d30f24e5-h88w4"] Oct 30 00:22:15.046325 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:15.046292 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d30f24e5-h88w4"] Oct 30 00:22:15.804886 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:15.804856 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7587-c250f6"] Oct 30 00:22:15.814701 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:15.814413 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7587-c250f6"] Oct 30 00:22:16.500422 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:16.500386 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-nxq9b"] Oct 30 00:22:16.501291 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:16.501274 5001 cpu_manager.go:414] "containerMap: removing container" podUID="67e1393f-a46e-4289-9715-04d738c00a7e" containerName="test-curl-7587-c250f6" Oct 30 00:22:16.501362 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:16.501292 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67e1393f-a46e-4289-9715-04d738c00a7e" containerName="test-curl-7587-c250f6" Oct 30 00:22:16.501429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:16.501417 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="67e1393f-a46e-4289-9715-04d738c00a7e" containerName="test-curl-7587-c250f6" Oct 30 00:22:16.508406 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:16.508383 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-nxq9b" Oct 30 00:22:16.526627 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:16.526595 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-nxq9b"] Oct 30 00:22:16.528310 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:16.528286 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:22:16.528310 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:16.528298 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:22:16.528471 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:16.528298 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:22:16.530593 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:16.530555 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:22:16.601774 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:16.601743 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ff6405-8fac-477b-aee9-08e629f42d8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-0847e8c0-nxq9b\" (UID: \"22ff6405-8fac-477b-aee9-08e629f42d8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-nxq9b" Oct 30 00:22:16.602043 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:16.602020 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ff6405-8fac-477b-aee9-08e629f42d8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-0847e8c0-nxq9b\" (UID: \"22ff6405-8fac-477b-aee9-08e629f42d8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-nxq9b" Oct 30 00:22:16.602159 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:16.602139 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f2bf\" (UniqueName: \"kubernetes.io/projected/22ff6405-8fac-477b-aee9-08e629f42d8c-kube-api-access-7f2bf\") pod \"maas-api-revoke-keys-e2e-mt-0847e8c0-nxq9b\" (UID: \"22ff6405-8fac-477b-aee9-08e629f42d8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-nxq9b" Oct 30 00:22:16.703598 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:16.703565 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ff6405-8fac-477b-aee9-08e629f42d8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-0847e8c0-nxq9b\" (UID: \"22ff6405-8fac-477b-aee9-08e629f42d8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-nxq9b" Oct 30 00:22:16.703778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:16.703637 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f2bf\" (UniqueName: \"kubernetes.io/projected/22ff6405-8fac-477b-aee9-08e629f42d8c-kube-api-access-7f2bf\") pod \"maas-api-revoke-keys-e2e-mt-0847e8c0-nxq9b\" (UID: \"22ff6405-8fac-477b-aee9-08e629f42d8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-nxq9b" Oct 30 00:22:16.703778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:16.703770 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ff6405-8fac-477b-aee9-08e629f42d8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-0847e8c0-nxq9b\" (UID: \"22ff6405-8fac-477b-aee9-08e629f42d8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-nxq9b" Oct 30 00:22:16.704487 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:16.704465 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ff6405-8fac-477b-aee9-08e629f42d8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-0847e8c0-nxq9b\" (UID: \"22ff6405-8fac-477b-aee9-08e629f42d8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-nxq9b" Oct 30 00:22:16.706103 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:16.706084 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ff6405-8fac-477b-aee9-08e629f42d8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-0847e8c0-nxq9b\" (UID: \"22ff6405-8fac-477b-aee9-08e629f42d8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-nxq9b" Oct 30 00:22:16.724273 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:16.724250 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f2bf\" (UniqueName: \"kubernetes.io/projected/22ff6405-8fac-477b-aee9-08e629f42d8c-kube-api-access-7f2bf\") pod \"maas-api-revoke-keys-e2e-mt-0847e8c0-nxq9b\" (UID: \"22ff6405-8fac-477b-aee9-08e629f42d8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-nxq9b" Oct 30 00:22:16.829203 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:16.829171 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-nxq9b" Oct 30 00:22:16.889764 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:16.887450 5001 generic.go:356] "Generic (PLEG): container finished" podID="1584ba56-ffef-49e1-8edd-399ca7b7f366" containerID="68562c9c74560a1ba68b6a3474a65cfdebcc91dd3793ba80d25924a386d06e26" exitCode=0 Oct 30 00:22:16.889764 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:16.887707 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7587-c250f6" podUID="67e1393f-a46e-4289-9715-04d738c00a7e" containerName="test-curl-7587-c250f6" containerID="cri-o://d71b5f2e087e9225a1bb75d197324d03023fb1b8e7b1f4b447ef8114d3bf70d2" gracePeriod=2 Oct 30 00:22:16.889764 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:16.887855 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" event={"ID":"1584ba56-ffef-49e1-8edd-399ca7b7f366","Type":"ContainerDied","Data":"68562c9c74560a1ba68b6a3474a65cfdebcc91dd3793ba80d25924a386d06e26"} Oct 30 00:22:16.889764 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:16.888408 5001 scope.go:122] "RemoveContainer" containerID="68562c9c74560a1ba68b6a3474a65cfdebcc91dd3793ba80d25924a386d06e26" Oct 30 00:22:16.891829 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:16.891791 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:22:16.891973 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:16.891889 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:22:16.892023 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:16.891967 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:22:16.924076 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:16.923735 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95b747aa-4fb0-4568-889c-40e44a2b5351" path="/var/lib/kubelet/pods/95b747aa-4fb0-4568-889c-40e44a2b5351/volumes" Oct 30 00:22:16.944500 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:22:16.941309 5001 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7587-c250f6\" is forbidden: User \"system:node:ip-10-0-143-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-120.ec2.internal' and this object" podUID="67e1393f-a46e-4289-9715-04d738c00a7e" pod="opendatahub/test-curl-7587-c250f6" Oct 30 00:22:17.138446 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:22:17.138413 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22ff6405_8fac_477b_aee9_08e629f42d8c.slice/crio-e5b23e25ff202bb9b5de2eb197287a2f4c58eb4a34d61584482fe61572cdf7ed WatchSource:0}: Error finding container e5b23e25ff202bb9b5de2eb197287a2f4c58eb4a34d61584482fe61572cdf7ed: Status 404 returned error can't find the container with id e5b23e25ff202bb9b5de2eb197287a2f4c58eb4a34d61584482fe61572cdf7ed Oct 30 00:22:17.138813 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:17.138784 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-nxq9b"] Oct 30 00:22:17.328172 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:22:17.328133 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:22:17.899132 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:17.899090 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-nxq9b" event={"ID":"22ff6405-8fac-477b-aee9-08e629f42d8c","Type":"ContainerStarted","Data":"0f47b897bf162924d8ad4be0a94889db4492558fba8db636ede7bcfdf63b8f59"} Oct 30 00:22:17.899565 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:17.899204 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-nxq9b" event={"ID":"22ff6405-8fac-477b-aee9-08e629f42d8c","Type":"ContainerStarted","Data":"e5b23e25ff202bb9b5de2eb197287a2f4c58eb4a34d61584482fe61572cdf7ed"} Oct 30 00:22:17.901384 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:17.901355 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" event={"ID":"1584ba56-ffef-49e1-8edd-399ca7b7f366","Type":"ContainerStarted","Data":"4ab18249e82ad6665281eaa53ea83fd1ba4eb61eee71e445a0700225c4d9d504"} Oct 30 00:22:17.947189 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:17.947047 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-nxq9b" podStartSLOduration=1.947028591 podStartE2EDuration="1.947028591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:22:17.945473956 +0000 UTC m=+1053.150975137" watchObservedRunningTime="2026-10-30 00:22:17.947028591 +0000 UTC m=+1053.152529772" Oct 30 00:22:18.387104 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:18.387070 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:22:18.388294 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:18.388251 5001 patch_prober.go:29] interesting pod/e2e-api-a-461609-openshift-default-86bb859445-gnfjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 00:22:18.388502 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:18.388327 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" podUID="1584ba56-ffef-49e1-8edd-399ca7b7f366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 00:22:18.927686 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:18.927656 5001 generic.go:356] "Generic (PLEG): container finished" podID="22ff6405-8fac-477b-aee9-08e629f42d8c" containerID="0f47b897bf162924d8ad4be0a94889db4492558fba8db636ede7bcfdf63b8f59" exitCode=0 Oct 30 00:22:18.939998 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:18.939952 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-nxq9b" event={"ID":"22ff6405-8fac-477b-aee9-08e629f42d8c","Type":"ContainerDied","Data":"0f47b897bf162924d8ad4be0a94889db4492558fba8db636ede7bcfdf63b8f59"} Oct 30 00:22:19.038939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:19.038226 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:22:19.312053 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:19.312017 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7587-c250f6" Oct 30 00:22:19.397375 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:19.397335 5001 patch_prober.go:29] interesting pod/e2e-api-a-461609-openshift-default-86bb859445-gnfjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 00:22:19.409256 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:19.409212 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" podUID="1584ba56-ffef-49e1-8edd-399ca7b7f366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 00:22:19.409856 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:19.399020 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67e1393f-a46e-4289-9715-04d738c00a7e-kube-api-access-9nps7\" (UniqueName: \"kubernetes.io/projected/67e1393f-a46e-4289-9715-04d738c00a7e-kube-api-access-9nps7\") pod \"67e1393f-a46e-4289-9715-04d738c00a7e\" (UID: \"67e1393f-a46e-4289-9715-04d738c00a7e\") " Oct 30 00:22:19.427188 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:19.427154 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e1393f-a46e-4289-9715-04d738c00a7e-kube-api-access-9nps7" pod "67e1393f-a46e-4289-9715-04d738c00a7e" (UID: "67e1393f-a46e-4289-9715-04d738c00a7e"). InnerVolumeSpecName "kube-api-access-9nps7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:22:19.559015 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:19.525739 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nps7\" (UniqueName: \"kubernetes.io/projected/67e1393f-a46e-4289-9715-04d738c00a7e-kube-api-access-9nps7\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:22:19.984328 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:19.984235 5001 generic.go:356] "Generic (PLEG): container finished" podID="67e1393f-a46e-4289-9715-04d738c00a7e" containerID="d71b5f2e087e9225a1bb75d197324d03023fb1b8e7b1f4b447ef8114d3bf70d2" exitCode=137 Oct 30 00:22:19.985500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:19.985476 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7587-c250f6" Oct 30 00:22:19.986709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:19.986320 5001 scope.go:122] "RemoveContainer" containerID="d71b5f2e087e9225a1bb75d197324d03023fb1b8e7b1f4b447ef8114d3bf70d2" Oct 30 00:22:20.003783 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:20.003761 5001 scope.go:122] "RemoveContainer" containerID="d71b5f2e087e9225a1bb75d197324d03023fb1b8e7b1f4b447ef8114d3bf70d2" Oct 30 00:22:20.004100 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:22:20.004078 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d71b5f2e087e9225a1bb75d197324d03023fb1b8e7b1f4b447ef8114d3bf70d2\": container with ID starting with d71b5f2e087e9225a1bb75d197324d03023fb1b8e7b1f4b447ef8114d3bf70d2 not found: ID does not exist" containerID="d71b5f2e087e9225a1bb75d197324d03023fb1b8e7b1f4b447ef8114d3bf70d2" Oct 30 00:22:20.004171 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:20.004112 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d71b5f2e087e9225a1bb75d197324d03023fb1b8e7b1f4b447ef8114d3bf70d2"} err="failed to get container status \"d71b5f2e087e9225a1bb75d197324d03023fb1b8e7b1f4b447ef8114d3bf70d2\": rpc error: code = NotFound desc = could not find container \"d71b5f2e087e9225a1bb75d197324d03023fb1b8e7b1f4b447ef8114d3bf70d2\": container with ID starting with d71b5f2e087e9225a1bb75d197324d03023fb1b8e7b1f4b447ef8114d3bf70d2 not found: ID does not exist" Oct 30 00:22:20.007376 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:22:20.007347 5001 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7587-c250f6\" is forbidden: User \"system:node:ip-10-0-143-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-120.ec2.internal' and this object" podUID="67e1393f-a46e-4289-9715-04d738c00a7e" pod="opendatahub/test-curl-7587-c250f6" Oct 30 00:22:20.390406 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:20.390347 5001 patch_prober.go:29] interesting pod/e2e-api-a-461609-openshift-default-86bb859445-gnfjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 00:22:20.390840 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:20.390804 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" podUID="1584ba56-ffef-49e1-8edd-399ca7b7f366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 00:22:20.870601 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:20.870145 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-nxq9b" Oct 30 00:22:20.883399 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:22:20.883205 5001 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7587-c250f6\" is forbidden: User \"system:node:ip-10-0-143-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-120.ec2.internal' and this object" podUID="67e1393f-a46e-4289-9715-04d738c00a7e" pod="opendatahub/test-curl-7587-c250f6" Oct 30 00:22:21.011974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:21.010814 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67e1393f-a46e-4289-9715-04d738c00a7e" path="/var/lib/kubelet/pods/67e1393f-a46e-4289-9715-04d738c00a7e/volumes" Oct 30 00:22:21.036620 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:21.036213 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-nxq9b" Oct 30 00:22:21.036620 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:21.036258 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-nxq9b" event={"ID":"22ff6405-8fac-477b-aee9-08e629f42d8c","Type":"ContainerDied","Data":"e5b23e25ff202bb9b5de2eb197287a2f4c58eb4a34d61584482fe61572cdf7ed"} Oct 30 00:22:21.036620 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:21.036532 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5b23e25ff202bb9b5de2eb197287a2f4c58eb4a34d61584482fe61572cdf7ed" Oct 30 00:22:21.090670 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:21.090639 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22ff6405-8fac-477b-aee9-08e629f42d8c-kube-api-access-7f2bf\" (UniqueName: \"kubernetes.io/projected/22ff6405-8fac-477b-aee9-08e629f42d8c-kube-api-access-7f2bf\") pod \"22ff6405-8fac-477b-aee9-08e629f42d8c\" (UID: \"22ff6405-8fac-477b-aee9-08e629f42d8c\") " Oct 30 00:22:21.090851 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:21.090836 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22ff6405-8fac-477b-aee9-08e629f42d8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ff6405-8fac-477b-aee9-08e629f42d8c-maas-api-xks-ca\") pod \"22ff6405-8fac-477b-aee9-08e629f42d8c\" (UID: \"22ff6405-8fac-477b-aee9-08e629f42d8c\") " Oct 30 00:22:21.091009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:21.090994 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22ff6405-8fac-477b-aee9-08e629f42d8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ff6405-8fac-477b-aee9-08e629f42d8c-maas-api-openshift-ca\") pod \"22ff6405-8fac-477b-aee9-08e629f42d8c\" (UID: \"22ff6405-8fac-477b-aee9-08e629f42d8c\") " Oct 30 00:22:21.091452 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:21.091426 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22ff6405-8fac-477b-aee9-08e629f42d8c-maas-api-openshift-ca" pod "22ff6405-8fac-477b-aee9-08e629f42d8c" (UID: "22ff6405-8fac-477b-aee9-08e629f42d8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:22:21.091765 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:21.091735 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ff6405-8fac-477b-aee9-08e629f42d8c-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:22:21.093514 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:21.093492 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22ff6405-8fac-477b-aee9-08e629f42d8c-maas-api-xks-ca" pod "22ff6405-8fac-477b-aee9-08e629f42d8c" (UID: "22ff6405-8fac-477b-aee9-08e629f42d8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:22:21.093825 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:21.093801 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22ff6405-8fac-477b-aee9-08e629f42d8c-kube-api-access-7f2bf" pod "22ff6405-8fac-477b-aee9-08e629f42d8c" (UID: "22ff6405-8fac-477b-aee9-08e629f42d8c"). InnerVolumeSpecName "kube-api-access-7f2bf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:22:21.193699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:21.193667 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f2bf\" (UniqueName: \"kubernetes.io/projected/22ff6405-8fac-477b-aee9-08e629f42d8c-kube-api-access-7f2bf\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:22:21.193905 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:21.193892 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ff6405-8fac-477b-aee9-08e629f42d8c-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:22:21.386901 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:21.386863 5001 patch_prober.go:29] interesting pod/e2e-api-a-461609-openshift-default-86bb859445-gnfjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 00:22:21.387100 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:21.386960 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" podUID="1584ba56-ffef-49e1-8edd-399ca7b7f366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 00:22:22.387597 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:22.387554 5001 patch_prober.go:29] interesting pod/e2e-api-a-461609-openshift-default-86bb859445-gnfjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 00:22:22.388223 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:22.387618 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" podUID="1584ba56-ffef-49e1-8edd-399ca7b7f366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 00:22:23.391017 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:23.390987 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:22:23.391994 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:23.391969 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:22:26.991938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:26.990664 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-fhf6b"] Oct 30 00:22:28.927935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:28.926988 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3fae6d-5xwgq"] Oct 30 00:22:29.068937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:29.068399 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3fae6d-5xwgq"] Oct 30 00:22:30.923665 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:30.923634 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72fdac6e-341f-4f95-a281-9137884a57d8" path="/var/lib/kubelet/pods/72fdac6e-341f-4f95-a281-9137884a57d8/volumes" Oct 30 00:22:31.991650 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:31.991610 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-fhf6b" podUID="d7a080fa-80b8-4542-b180-40a65c8bb758" containerName="payload-pre-processing" containerID="cri-o://edb38dd57203af7c4cabdc9e4c5c0e1fb5ff2a7e1b6d2d071b4f1d34e9c7f6bf" gracePeriod=25 Oct 30 00:22:32.437505 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:32.437474 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-fhf6b" Oct 30 00:22:32.439593 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:32.439547 5001 generic.go:356] "Generic (PLEG): container finished" podID="d7a080fa-80b8-4542-b180-40a65c8bb758" containerID="edb38dd57203af7c4cabdc9e4c5c0e1fb5ff2a7e1b6d2d071b4f1d34e9c7f6bf" exitCode=0 Oct 30 00:22:32.439950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:32.439926 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-fhf6b" event={"ID":"d7a080fa-80b8-4542-b180-40a65c8bb758","Type":"ContainerDied","Data":"edb38dd57203af7c4cabdc9e4c5c0e1fb5ff2a7e1b6d2d071b4f1d34e9c7f6bf"} Oct 30 00:22:32.440092 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:32.440013 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-fhf6b" event={"ID":"d7a080fa-80b8-4542-b180-40a65c8bb758","Type":"ContainerDied","Data":"b21c9f22e48d58d7dfd9e5bf5d796e169924971ddeb7efc15808ef5d572d30d3"} Oct 30 00:22:32.440092 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:32.440045 5001 scope.go:122] "RemoveContainer" containerID="edb38dd57203af7c4cabdc9e4c5c0e1fb5ff2a7e1b6d2d071b4f1d34e9c7f6bf" Oct 30 00:22:32.440221 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:32.440208 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-fhf6b" Oct 30 00:22:32.452558 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:32.452525 5001 scope.go:122] "RemoveContainer" containerID="edb38dd57203af7c4cabdc9e4c5c0e1fb5ff2a7e1b6d2d071b4f1d34e9c7f6bf" Oct 30 00:22:32.452840 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:22:32.452820 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edb38dd57203af7c4cabdc9e4c5c0e1fb5ff2a7e1b6d2d071b4f1d34e9c7f6bf\": container with ID starting with edb38dd57203af7c4cabdc9e4c5c0e1fb5ff2a7e1b6d2d071b4f1d34e9c7f6bf not found: ID does not exist" containerID="edb38dd57203af7c4cabdc9e4c5c0e1fb5ff2a7e1b6d2d071b4f1d34e9c7f6bf" Oct 30 00:22:32.452893 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:32.452853 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb38dd57203af7c4cabdc9e4c5c0e1fb5ff2a7e1b6d2d071b4f1d34e9c7f6bf"} err="failed to get container status \"edb38dd57203af7c4cabdc9e4c5c0e1fb5ff2a7e1b6d2d071b4f1d34e9c7f6bf\": rpc error: code = NotFound desc = could not find container \"edb38dd57203af7c4cabdc9e4c5c0e1fb5ff2a7e1b6d2d071b4f1d34e9c7f6bf\": container with ID starting with edb38dd57203af7c4cabdc9e4c5c0e1fb5ff2a7e1b6d2d071b4f1d34e9c7f6bf not found: ID does not exist" Oct 30 00:22:32.557393 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:32.557347 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7a080fa-80b8-4542-b180-40a65c8bb758-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a080fa-80b8-4542-b180-40a65c8bb758-plugins-config-volume\") pod \"d7a080fa-80b8-4542-b180-40a65c8bb758\" (UID: \"d7a080fa-80b8-4542-b180-40a65c8bb758\") " Oct 30 00:22:32.557783 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:32.557759 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7a080fa-80b8-4542-b180-40a65c8bb758-plugins-config-volume" pod "d7a080fa-80b8-4542-b180-40a65c8bb758" (UID: "d7a080fa-80b8-4542-b180-40a65c8bb758"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:22:32.557846 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:32.557795 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7a080fa-80b8-4542-b180-40a65c8bb758-kube-api-access-shct4\" (UniqueName: \"kubernetes.io/projected/d7a080fa-80b8-4542-b180-40a65c8bb758-kube-api-access-shct4\") pod \"d7a080fa-80b8-4542-b180-40a65c8bb758\" (UID: \"d7a080fa-80b8-4542-b180-40a65c8bb758\") " Oct 30 00:22:32.558364 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:32.558346 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a080fa-80b8-4542-b180-40a65c8bb758-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:22:32.559877 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:32.559856 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7a080fa-80b8-4542-b180-40a65c8bb758-kube-api-access-shct4" pod "d7a080fa-80b8-4542-b180-40a65c8bb758" (UID: "d7a080fa-80b8-4542-b180-40a65c8bb758"). InnerVolumeSpecName "kube-api-access-shct4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:22:32.660010 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:32.659906 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shct4\" (UniqueName: \"kubernetes.io/projected/d7a080fa-80b8-4542-b180-40a65c8bb758-kube-api-access-shct4\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:22:32.784625 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:32.784591 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-fhf6b"] Oct 30 00:22:32.793779 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:32.793753 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0847e8c0-cb9b56d67-fhf6b"] Oct 30 00:22:32.920944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:32.920859 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7a080fa-80b8-4542-b180-40a65c8bb758" path="/var/lib/kubelet/pods/d7a080fa-80b8-4542-b180-40a65c8bb758/volumes" Oct 30 00:22:35.493982 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:35.475259 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht"] Oct 30 00:22:35.495611 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:35.495321 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" podUID="f42670c1-7716-46d8-9bc3-3f06cf8caa8a" containerName="istio-proxy" containerID="cri-o://e7ba467a236ec1b1538f4ad87076753316038898b6856dbdcb97803816025c94" gracePeriod=30 Oct 30 00:22:36.062243 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:36.062209 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7417dc-gncfw"] Oct 30 00:22:36.063275 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:36.063248 5001 cpu_manager.go:414] "containerMap: removing container" podUID="22ff6405-8fac-477b-aee9-08e629f42d8c" containerName="revoke-keys" Oct 30 00:22:36.063275 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:36.063275 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22ff6405-8fac-477b-aee9-08e629f42d8c" containerName="revoke-keys" Oct 30 00:22:36.063476 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:36.063329 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d7a080fa-80b8-4542-b180-40a65c8bb758" containerName="payload-pre-processing" Oct 30 00:22:36.063476 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:36.063339 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7a080fa-80b8-4542-b180-40a65c8bb758" containerName="payload-pre-processing" Oct 30 00:22:36.063590 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:36.063527 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="22ff6405-8fac-477b-aee9-08e629f42d8c" containerName="revoke-keys" Oct 30 00:22:36.063590 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:36.063549 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7a080fa-80b8-4542-b180-40a65c8bb758" containerName="payload-pre-processing" Oct 30 00:22:36.069740 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:36.069721 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7417dc-gncfw" Oct 30 00:22:36.082198 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:36.082175 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:22:36.082317 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:36.082282 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:22:36.091902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:36.091883 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:22:36.092014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:36.091953 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:22:36.169743 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:36.169708 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ca0c60-e7de-43f7-8cac-db4372fa30d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7417dc-gncfw\" (UID: \"08ca0c60-e7de-43f7-8cac-db4372fa30d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7417dc-gncfw" Oct 30 00:22:36.169743 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:36.169745 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zwww\" (UniqueName: \"kubernetes.io/projected/08ca0c60-e7de-43f7-8cac-db4372fa30d2-kube-api-access-4zwww\") pod \"maas-api-revoke-keys-e2e-api-b-7417dc-gncfw\" (UID: \"08ca0c60-e7de-43f7-8cac-db4372fa30d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7417dc-gncfw" Oct 30 00:22:36.169963 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:36.169904 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ca0c60-e7de-43f7-8cac-db4372fa30d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7417dc-gncfw\" (UID: \"08ca0c60-e7de-43f7-8cac-db4372fa30d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7417dc-gncfw" Oct 30 00:22:36.271463 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:36.271428 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ca0c60-e7de-43f7-8cac-db4372fa30d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7417dc-gncfw\" (UID: \"08ca0c60-e7de-43f7-8cac-db4372fa30d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7417dc-gncfw" Oct 30 00:22:36.271463 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:36.271463 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zwww\" (UniqueName: \"kubernetes.io/projected/08ca0c60-e7de-43f7-8cac-db4372fa30d2-kube-api-access-4zwww\") pod \"maas-api-revoke-keys-e2e-api-b-7417dc-gncfw\" (UID: \"08ca0c60-e7de-43f7-8cac-db4372fa30d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7417dc-gncfw" Oct 30 00:22:36.271670 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:36.271564 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ca0c60-e7de-43f7-8cac-db4372fa30d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7417dc-gncfw\" (UID: \"08ca0c60-e7de-43f7-8cac-db4372fa30d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7417dc-gncfw" Oct 30 00:22:36.272228 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:36.272202 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ca0c60-e7de-43f7-8cac-db4372fa30d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7417dc-gncfw\" (UID: \"08ca0c60-e7de-43f7-8cac-db4372fa30d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7417dc-gncfw" Oct 30 00:22:36.274044 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:36.274019 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ca0c60-e7de-43f7-8cac-db4372fa30d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7417dc-gncfw\" (UID: \"08ca0c60-e7de-43f7-8cac-db4372fa30d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7417dc-gncfw" Oct 30 00:22:36.292231 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:36.292203 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zwww\" (UniqueName: \"kubernetes.io/projected/08ca0c60-e7de-43f7-8cac-db4372fa30d2-kube-api-access-4zwww\") pod \"maas-api-revoke-keys-e2e-api-b-7417dc-gncfw\" (UID: \"08ca0c60-e7de-43f7-8cac-db4372fa30d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7417dc-gncfw" Oct 30 00:22:36.386095 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:36.386028 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7417dc-gncfw" Oct 30 00:22:40.817800 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:40.812523 5001 generic.go:356] "Generic (PLEG): container finished" podID="f42670c1-7716-46d8-9bc3-3f06cf8caa8a" containerID="e7ba467a236ec1b1538f4ad87076753316038898b6856dbdcb97803816025c94" exitCode=0 Oct 30 00:22:40.817800 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:40.812957 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" event={"ID":"f42670c1-7716-46d8-9bc3-3f06cf8caa8a","Type":"ContainerDied","Data":"e7ba467a236ec1b1538f4ad87076753316038898b6856dbdcb97803816025c94"} Oct 30 00:22:40.817800 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:40.813051 5001 scope.go:122] "RemoveContainer" containerID="5228870d920b4c45c868cb243140a6e055def8b47c6d8277afa434d652ba9250" Oct 30 00:22:41.383050 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:41.377071 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:41.418936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:41.418025 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:22:41.439932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:41.438075 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:22:41.572839 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:41.572790 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 00:22:41.781564 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:41.771528 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:22:41.881263 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:41.870054 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:41.881263 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:41.870584 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": context deadline exceeded" Oct 30 00:22:41.881263 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:41.870895 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" podUID="b3e66631-9562-4bb5-b005-1fc179a8582d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.135:8000/ready\": context deadline exceeded" Oct 30 00:22:41.932176 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:41.931661 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:22:42.216956 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.189596 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13496902-jv9sj"] Oct 30 00:22:42.216956 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.205595 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:42.244146 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.243710 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:22:42.248267 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.247404 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:42.288990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.287017 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13496902-jv9sj"] Oct 30 00:22:42.352454 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.350444 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istio-podinfo\") pod \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " Oct 30 00:22:42.352454 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.350508 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-workload-certs\") pod \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " Oct 30 00:22:42.352454 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.350577 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istio-data\") pod \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " Oct 30 00:22:42.352454 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.350651 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istio-token\" (UniqueName: \"kubernetes.io/projected/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istio-token\") pod \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " Oct 30 00:22:42.352454 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.350757 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-credential-socket\") pod \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " Oct 30 00:22:42.352454 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.350808 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-workload-socket\") pod \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " Oct 30 00:22:42.352454 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.351096 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istiod-ca-cert\") pod \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " Oct 30 00:22:42.352454 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.351189 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istio-envoy\") pod \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " Oct 30 00:22:42.352454 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.351219 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-kube-api-access-66wk5\" (UniqueName: \"kubernetes.io/projected/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-kube-api-access-66wk5\") pod \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\" (UID: \"f42670c1-7716-46d8-9bc3-3f06cf8caa8a\") " Oct 30 00:22:42.358230 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.358194 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-workload-socket" pod "f42670c1-7716-46d8-9bc3-3f06cf8caa8a" (UID: "f42670c1-7716-46d8-9bc3-3f06cf8caa8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:22:42.358613 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.358591 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-credential-socket" pod "f42670c1-7716-46d8-9bc3-3f06cf8caa8a" (UID: "f42670c1-7716-46d8-9bc3-3f06cf8caa8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:22:42.359053 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.359033 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istiod-ca-cert" pod "f42670c1-7716-46d8-9bc3-3f06cf8caa8a" (UID: "f42670c1-7716-46d8-9bc3-3f06cf8caa8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:22:42.359964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.359698 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-workload-certs" pod "f42670c1-7716-46d8-9bc3-3f06cf8caa8a" (UID: "f42670c1-7716-46d8-9bc3-3f06cf8caa8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:22:42.361157 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.361136 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istio-data" pod "f42670c1-7716-46d8-9bc3-3f06cf8caa8a" (UID: "f42670c1-7716-46d8-9bc3-3f06cf8caa8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:22:42.376784 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.376738 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istio-token" pod "f42670c1-7716-46d8-9bc3-3f06cf8caa8a" (UID: "f42670c1-7716-46d8-9bc3-3f06cf8caa8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:22:42.377942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.377877 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 00:22:42.379056 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.378784 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:22:42.401075 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.401048 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:22:42.424506 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.424472 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-kube-api-access-66wk5" pod "f42670c1-7716-46d8-9bc3-3f06cf8caa8a" (UID: "f42670c1-7716-46d8-9bc3-3f06cf8caa8a"). InnerVolumeSpecName "kube-api-access-66wk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:22:42.450665 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.450432 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istio-podinfo" pod "f42670c1-7716-46d8-9bc3-3f06cf8caa8a" (UID: "f42670c1-7716-46d8-9bc3-3f06cf8caa8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:22:42.464966 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.454239 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:22:42.464966 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.454271 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:22:42.464966 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.454288 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:22:42.464966 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.454302 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:22:42.464966 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.454316 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66wk5\" (UniqueName: \"kubernetes.io/projected/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-kube-api-access-66wk5\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:22:42.464966 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.454329 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:22:42.464966 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.454345 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:22:42.464966 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.454359 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:22:42.464966 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.462049 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istio-envoy" pod "f42670c1-7716-46d8-9bc3-3f06cf8caa8a" (UID: "f42670c1-7716-46d8-9bc3-3f06cf8caa8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:22:42.563698 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.562813 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f42670c1-7716-46d8-9bc3-3f06cf8caa8a-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:22:42.734936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.731263 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:42.781140 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:42.781090 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:22:43.203936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:43.201486 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d94a02d-a312-4c91-a1c7-bb09907fc5ae" path="/var/lib/kubelet/pods/7d94a02d-a312-4c91-a1c7-bb09907fc5ae/volumes" Oct 30 00:22:43.575703 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:43.575672 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:22:43.714123 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:43.714018 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:43.747735 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:43.743099 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" Oct 30 00:22:43.747735 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:43.744596 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht" event={"ID":"f42670c1-7716-46d8-9bc3-3f06cf8caa8a","Type":"ContainerDied","Data":"65d96276fe8706cd9dc26b290762928fbe422a2c024b1206faefa31549019cb4"} Oct 30 00:22:43.747735 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:43.744689 5001 scope.go:122] "RemoveContainer" containerID="e7ba467a236ec1b1538f4ad87076753316038898b6856dbdcb97803816025c94" Oct 30 00:22:43.804320 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:43.804171 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:43.895827 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:43.877109 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" podUID="b3e66631-9562-4bb5-b005-1fc179a8582d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.135:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:44.201937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:44.201724 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" podUID="b3e66631-9562-4bb5-b005-1fc179a8582d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.135:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:44.221033 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:44.220980 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht"] Oct 30 00:22:44.229089 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:44.229039 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:44.379021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:44.376636 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-3fae6d-openshift-default-c696f6bbc-hvzht"] Oct 30 00:22:44.404584 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:44.404534 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:44.418623 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:44.418580 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:44.456067 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:44.456020 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:44.495938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:44.475469 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35b82218-88cpk"] Oct 30 00:22:44.497422 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:44.497384 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f42670c1-7716-46d8-9bc3-3f06cf8caa8a" containerName="istio-proxy" Oct 30 00:22:44.497596 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:44.497579 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f42670c1-7716-46d8-9bc3-3f06cf8caa8a" containerName="istio-proxy" Oct 30 00:22:44.512433 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:44.512397 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f42670c1-7716-46d8-9bc3-3f06cf8caa8a" containerName="istio-proxy" Oct 30 00:22:44.523940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:44.520188 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b82218-88cpk" Oct 30 00:22:44.541382 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:44.541344 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f086c625-6fda-4000-9abf-1055f720d140-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35b82218-88cpk\" (UID: \"f086c625-6fda-4000-9abf-1055f720d140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b82218-88cpk" Oct 30 00:22:44.541692 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:44.541667 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdrpw\" (UniqueName: \"kubernetes.io/projected/f086c625-6fda-4000-9abf-1055f720d140-kube-api-access-fdrpw\") pod \"maas-api-revoke-sub-35b82218-88cpk\" (UID: \"f086c625-6fda-4000-9abf-1055f720d140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b82218-88cpk" Oct 30 00:22:44.541861 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:44.541843 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f086c625-6fda-4000-9abf-1055f720d140-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35b82218-88cpk\" (UID: \"f086c625-6fda-4000-9abf-1055f720d140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b82218-88cpk" Oct 30 00:22:44.605934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:44.589840 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:44.662993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:44.658937 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f086c625-6fda-4000-9abf-1055f720d140-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35b82218-88cpk\" (UID: \"f086c625-6fda-4000-9abf-1055f720d140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b82218-88cpk" Oct 30 00:22:44.666954 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:44.663327 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdrpw\" (UniqueName: \"kubernetes.io/projected/f086c625-6fda-4000-9abf-1055f720d140-kube-api-access-fdrpw\") pod \"maas-api-revoke-sub-35b82218-88cpk\" (UID: \"f086c625-6fda-4000-9abf-1055f720d140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b82218-88cpk" Oct 30 00:22:44.683087 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:44.683046 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f086c625-6fda-4000-9abf-1055f720d140-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35b82218-88cpk\" (UID: \"f086c625-6fda-4000-9abf-1055f720d140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b82218-88cpk" Oct 30 00:22:44.684499 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:44.684470 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f086c625-6fda-4000-9abf-1055f720d140-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35b82218-88cpk\" (UID: \"f086c625-6fda-4000-9abf-1055f720d140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b82218-88cpk" Oct 30 00:22:44.722951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:44.695632 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f086c625-6fda-4000-9abf-1055f720d140-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35b82218-88cpk\" (UID: \"f086c625-6fda-4000-9abf-1055f720d140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b82218-88cpk" Oct 30 00:22:44.722951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:44.704738 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdrpw\" (UniqueName: \"kubernetes.io/projected/f086c625-6fda-4000-9abf-1055f720d140-kube-api-access-fdrpw\") pod \"maas-api-revoke-sub-35b82218-88cpk\" (UID: \"f086c625-6fda-4000-9abf-1055f720d140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b82218-88cpk" Oct 30 00:22:44.723433 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:44.723388 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:44.724836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:44.724807 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:22:44.814944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:44.814217 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:44.828956 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:44.815275 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:22:44.829142 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:44.828982 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 30 00:22:44.879078 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:44.879034 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" podUID="b3e66631-9562-4bb5-b005-1fc179a8582d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.135:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:44.880525 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:44.880499 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" Oct 30 00:22:44.966290 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:44.966197 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b82218-88cpk" Oct 30 00:22:45.188933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:45.179233 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ff75p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:22:45.188933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:45.179310 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ff75p" podUID="40f44089-5854-4325-820f-b3481352eb28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:45.191168 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:45.191131 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f42670c1-7716-46d8-9bc3-3f06cf8caa8a" path="/var/lib/kubelet/pods/f42670c1-7716-46d8-9bc3-3f06cf8caa8a/volumes" Oct 30 00:22:45.290245 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:45.290215 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-461609-g52q5"] Oct 30 00:22:45.292001 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:45.291973 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f42670c1-7716-46d8-9bc3-3f06cf8caa8a" containerName="istio-proxy" Oct 30 00:22:45.292170 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:45.292155 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f42670c1-7716-46d8-9bc3-3f06cf8caa8a" containerName="istio-proxy" Oct 30 00:22:45.292500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:45.292482 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f42670c1-7716-46d8-9bc3-3f06cf8caa8a" containerName="istio-proxy" Oct 30 00:22:45.434934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:45.406332 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:22:45.434934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:45.422758 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:45.434934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:45.431234 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:22:45.434934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:45.431301 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:22:45.434934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:45.432731 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-461609-g52q5" Oct 30 00:22:45.481292 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:45.462930 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:45.487539 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:45.487506 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:22:45.636983 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:45.598058 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf5f5d9e-2d6f-4861-9279-ce514f89dd86-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-461609-g52q5\" (UID: \"cf5f5d9e-2d6f-4861-9279-ce514f89dd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-461609-g52q5" Oct 30 00:22:45.645088 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:45.645051 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5r4c\" (UniqueName: \"kubernetes.io/projected/cf5f5d9e-2d6f-4861-9279-ce514f89dd86-kube-api-access-m5r4c\") pod \"maas-api-revoke-keys-e2e-api-a-461609-g52q5\" (UID: \"cf5f5d9e-2d6f-4861-9279-ce514f89dd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-461609-g52q5" Oct 30 00:22:45.653101 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:45.652438 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf5f5d9e-2d6f-4861-9279-ce514f89dd86-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-461609-g52q5\" (UID: \"cf5f5d9e-2d6f-4861-9279-ce514f89dd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-461609-g52q5" Oct 30 00:22:45.653101 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:45.637324 5001 scope.go:122] "RemoveContainer" containerID="b87a6e349eb5724d27cab5601555fd0c0affbc3008df7277dd054b8017bb4c60" Oct 30 00:22:45.746599 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:45.603136 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 00:22:45.792304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:45.728056 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:45.793071 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:45.793044 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:22:45.793544 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:45.788248 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf5f5d9e-2d6f-4861-9279-ce514f89dd86-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-461609-g52q5\" (UID: \"cf5f5d9e-2d6f-4861-9279-ce514f89dd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-461609-g52q5" Oct 30 00:22:45.804303 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:45.804270 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf5f5d9e-2d6f-4861-9279-ce514f89dd86-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-461609-g52q5\" (UID: \"cf5f5d9e-2d6f-4861-9279-ce514f89dd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-461609-g52q5" Oct 30 00:22:45.804507 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:45.804489 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5r4c\" (UniqueName: \"kubernetes.io/projected/cf5f5d9e-2d6f-4861-9279-ce514f89dd86-kube-api-access-m5r4c\") pod \"maas-api-revoke-keys-e2e-api-a-461609-g52q5\" (UID: \"cf5f5d9e-2d6f-4861-9279-ce514f89dd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-461609-g52q5" Oct 30 00:22:45.805835 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:45.805810 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf5f5d9e-2d6f-4861-9279-ce514f89dd86-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-461609-g52q5\" (UID: \"cf5f5d9e-2d6f-4861-9279-ce514f89dd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-461609-g52q5" Oct 30 00:22:45.817398 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:45.817363 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:45.836935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:45.832034 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:45.836935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:45.836131 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:22:45.909988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:45.883814 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf5f5d9e-2d6f-4861-9279-ce514f89dd86-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-461609-g52q5\" (UID: \"cf5f5d9e-2d6f-4861-9279-ce514f89dd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-461609-g52q5" Oct 30 00:22:45.909988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:45.906870 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" podUID="b3e66631-9562-4bb5-b005-1fc179a8582d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.135:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:45.917516 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:45.917139 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:22:46.076963 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:46.066754 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5r4c\" (UniqueName: \"kubernetes.io/projected/cf5f5d9e-2d6f-4861-9279-ce514f89dd86-kube-api-access-m5r4c\") pod \"maas-api-revoke-keys-e2e-api-a-461609-g52q5\" (UID: \"cf5f5d9e-2d6f-4861-9279-ce514f89dd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-461609-g52q5" Oct 30 00:22:46.319465 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:46.310067 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:46.354967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:46.326421 5001 scope.go:122] "RemoveContainer" containerID="f625604a8d269b50f9fc112ed41688f3616dec972d54a3f4a850973a666e7e68" Oct 30 00:22:46.354967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:46.337422 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-461609-g52q5" Oct 30 00:22:46.491111 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:46.491068 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:22:46.491856 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:46.491804 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 00:22:46.592903 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:46.592812 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:22:46.929994 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:46.908896 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" podUID="b3e66631-9562-4bb5-b005-1fc179a8582d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.135:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:46.929994 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:46.925945 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:46.929994 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:46.926038 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:46.929994 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:46.927395 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 30 00:22:47.009974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:46.990766 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:47.076433 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:47.076398 5001 scope.go:122] "RemoveContainer" containerID="2a7437a334123b1c3b1c87b92aed6e64d3d0d6877952d29e1d42c623344804d9" Oct 30 00:22:47.084390 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:47.084347 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:47.336117 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:47.336071 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:47.394935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:47.370516 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:22:47.401778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:47.401742 5001 scope.go:122] "RemoveContainer" containerID="53246042e69856a9043b1257163f17639ad44a4d09ac4a00319709b4dee277cd" Oct 30 00:22:47.491611 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:47.491567 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:47.558641 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:47.558506 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:47.603462 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:47.603434 5001 scope.go:122] "RemoveContainer" containerID="c3ead80eafae4db22c8ee1ae97dc33ebe657e5556fe8b389679325f01885cb1a" Oct 30 00:22:47.851337 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:47.851054 5001 scope.go:122] "RemoveContainer" containerID="70f766da62595ccde16900c36c0bf36c6e30c5b17046f62c8498d9cd275cae99" Oct 30 00:22:47.926388 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:47.926285 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:47.927582 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:47.927547 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:47.956082 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:47.955013 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" podUID="b3e66631-9562-4bb5-b005-1fc179a8582d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.135:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:47.956082 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:47.955769 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:47.964606 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:22:47.964537 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:22:48.019001 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:48.018043 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:48.379936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:48.379004 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:48.468938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:48.454610 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:48.509931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:48.509090 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:48.565037 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:48.559742 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:48.955937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:48.930743 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:22:48.956641 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:48.936145 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:48.957369 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:48.939969 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:22:48.982931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:48.962886 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" podUID="b3e66631-9562-4bb5-b005-1fc179a8582d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.135:8000/ready\": context deadline exceeded" Oct 30 00:22:48.982931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:48.982038 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:49.035047 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:49.014014 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:49.035047 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:49.021761 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:22:49.035047 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:49.021815 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:22:49.226995 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:49.223412 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:22:49.357935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:49.357035 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": context deadline exceeded" Oct 30 00:22:49.397053 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:49.397006 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:49.399516 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:49.399491 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:22:49.568987 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:49.526451 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:49.746457 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:49.744423 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" Oct 30 00:22:49.969078 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:49.967019 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:49.969078 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:49.967405 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:22:49.969078 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:49.967979 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:22:50.025931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:50.017049 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:50.192262 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:50.192231 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:22:50.232157 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:50.231102 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:22:50.318932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:50.309550 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:22:50.407163 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:50.400428 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:22:50.407163 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:50.402072 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:22:53.012084 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:53.012033 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:22:53.012734 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:53.012116 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" podUID="4bfbc166-3dbf-4aae-96f5-515f59917846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:56.279132 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:56.279069 5001 patch_prober.go:29] interesting pod/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:22:56.280067 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:56.279175 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" podUID="a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:22:58.503639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:57.665087 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e2a011-tw9s7"] Oct 30 00:22:58.503639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:57.683310 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e2a011-tw9s7" Oct 30 00:22:58.503639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:57.787369 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62zwx\" (UniqueName: \"kubernetes.io/projected/b8c83212-4e77-49a1-a39c-180047d688c0-kube-api-access-62zwx\") pod \"maas-api-revoke-keys-e2e-api-a-e2a011-tw9s7\" (UID: \"b8c83212-4e77-49a1-a39c-180047d688c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e2a011-tw9s7" Oct 30 00:22:58.503639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:57.787514 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c83212-4e77-49a1-a39c-180047d688c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e2a011-tw9s7\" (UID: \"b8c83212-4e77-49a1-a39c-180047d688c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e2a011-tw9s7" Oct 30 00:22:58.503639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:57.787584 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c83212-4e77-49a1-a39c-180047d688c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e2a011-tw9s7\" (UID: \"b8c83212-4e77-49a1-a39c-180047d688c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e2a011-tw9s7" Oct 30 00:22:58.503639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:57.889246 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62zwx\" (UniqueName: \"kubernetes.io/projected/b8c83212-4e77-49a1-a39c-180047d688c0-kube-api-access-62zwx\") pod \"maas-api-revoke-keys-e2e-api-a-e2a011-tw9s7\" (UID: \"b8c83212-4e77-49a1-a39c-180047d688c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e2a011-tw9s7" Oct 30 00:22:58.503639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:57.889338 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c83212-4e77-49a1-a39c-180047d688c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e2a011-tw9s7\" (UID: \"b8c83212-4e77-49a1-a39c-180047d688c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e2a011-tw9s7" Oct 30 00:22:58.503639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:57.889371 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c83212-4e77-49a1-a39c-180047d688c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e2a011-tw9s7\" (UID: \"b8c83212-4e77-49a1-a39c-180047d688c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e2a011-tw9s7" Oct 30 00:22:58.503639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:57.890056 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c83212-4e77-49a1-a39c-180047d688c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e2a011-tw9s7\" (UID: \"b8c83212-4e77-49a1-a39c-180047d688c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e2a011-tw9s7" Oct 30 00:22:58.503639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:57.891839 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c83212-4e77-49a1-a39c-180047d688c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e2a011-tw9s7\" (UID: \"b8c83212-4e77-49a1-a39c-180047d688c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e2a011-tw9s7" Oct 30 00:22:58.503639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:57.910027 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62zwx\" (UniqueName: \"kubernetes.io/projected/b8c83212-4e77-49a1-a39c-180047d688c0-kube-api-access-62zwx\") pod \"maas-api-revoke-keys-e2e-api-a-e2a011-tw9s7\" (UID: \"b8c83212-4e77-49a1-a39c-180047d688c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e2a011-tw9s7" Oct 30 00:22:58.503639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:58.017498 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e2a011-tw9s7" Oct 30 00:22:58.503639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:58.196661 5001 generic.go:356] "Generic (PLEG): container finished" podID="4bfbc166-3dbf-4aae-96f5-515f59917846" containerID="af07da8cc8da63de4de9af460ea429fc7f10f344885d5ee21a8d9d1a291b8d3d" exitCode=0 Oct 30 00:22:58.503639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:58.196721 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" event={"ID":"4bfbc166-3dbf-4aae-96f5-515f59917846","Type":"ContainerDied","Data":"af07da8cc8da63de4de9af460ea429fc7f10f344885d5ee21a8d9d1a291b8d3d"} Oct 30 00:22:58.503639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:58.197181 5001 scope.go:122] "RemoveContainer" containerID="af07da8cc8da63de4de9af460ea429fc7f10f344885d5ee21a8d9d1a291b8d3d" Oct 30 00:22:58.503639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:58.198726 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:22:58.503639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:58.198797 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:22:58.503639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:58.198833 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:22:59.227042 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:59.208100 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" event={"ID":"4bfbc166-3dbf-4aae-96f5-515f59917846","Type":"ContainerStarted","Data":"c4f68de95ea8a0fe7c2ab9ab7df309f72eb2d21accc0ed06ee4d1a7700da57e9"} Oct 30 00:22:59.235944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:59.235252 5001 generic.go:356] "Generic (PLEG): container finished" podID="ed3d03e4-5550-40bc-8c00-48c3c7927fa1" containerID="dc5d7805cd01f09ae7d448989b3fcd91eddda4a0896f21dd546d223c3494e577" exitCode=0 Oct 30 00:22:59.235944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:59.235379 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" event={"ID":"ed3d03e4-5550-40bc-8c00-48c3c7927fa1","Type":"ContainerDied","Data":"dc5d7805cd01f09ae7d448989b3fcd91eddda4a0896f21dd546d223c3494e577"} Oct 30 00:22:59.235944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:59.235871 5001 scope.go:122] "RemoveContainer" containerID="dc5d7805cd01f09ae7d448989b3fcd91eddda4a0896f21dd546d223c3494e577" Oct 30 00:22:59.239237 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:59.238209 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:22:59.239237 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:59.238293 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:22:59.239237 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:59.238338 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:22:59.239237 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:59.238598 5001 generic.go:356] "Generic (PLEG): container finished" podID="a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7" containerID="d8349c31038fd354f846c7f57f925a6d777d8112a1b5b84ce9d9c75c0a5f42ad" exitCode=0 Oct 30 00:22:59.239237 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:59.238681 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" event={"ID":"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7","Type":"ContainerDied","Data":"d8349c31038fd354f846c7f57f925a6d777d8112a1b5b84ce9d9c75c0a5f42ad"} Oct 30 00:22:59.239237 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:59.239205 5001 scope.go:122] "RemoveContainer" containerID="d8349c31038fd354f846c7f57f925a6d777d8112a1b5b84ce9d9c75c0a5f42ad" Oct 30 00:22:59.241051 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:59.241018 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:22:59.241179 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:59.241123 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:22:59.241229 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:59.241179 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:22:59.440976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:59.437867 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:22:59.440976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:59.438049 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 00:22:59.440976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:22:59.438117 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" podUID="4bfbc166-3dbf-4aae-96f5-515f59917846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 00:23:00.289944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:00.288306 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" event={"ID":"ed3d03e4-5550-40bc-8c00-48c3c7927fa1","Type":"ContainerStarted","Data":"236dd29a33f1bc6f9edc1a3037b646e0ee769237581b5c6333a209b2877f3f3f"} Oct 30 00:23:00.293017 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:00.292989 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" event={"ID":"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7","Type":"ContainerStarted","Data":"d6025520905dea45a36fdf8a8ec0387ffbff91650d87c4bb7af7c87b2e06524c"} Oct 30 00:23:00.434935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:00.434323 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 00:23:00.434935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:00.434389 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" podUID="4bfbc166-3dbf-4aae-96f5-515f59917846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 00:23:00.876937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:00.876306 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:23:00.888181 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:00.888120 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 00:23:00.888387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:00.888207 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" podUID="ed3d03e4-5550-40bc-8c00-48c3c7927fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 00:23:01.002070 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:01.002044 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:23:01.002316 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:01.002238 5001 patch_prober.go:29] interesting pod/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 00:23:01.002448 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:01.002431 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" podUID="a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 00:23:01.469937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:01.466346 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 00:23:01.469937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:01.466436 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" podUID="4bfbc166-3dbf-4aae-96f5-515f59917846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 00:23:01.868934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:01.868338 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 00:23:01.868934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:01.868406 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" podUID="ed3d03e4-5550-40bc-8c00-48c3c7927fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 00:23:02.003454 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:02.003171 5001 patch_prober.go:29] interesting pod/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 00:23:02.003454 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:02.003255 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" podUID="a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 00:23:02.448050 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:02.445370 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 00:23:02.448050 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:02.445449 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" podUID="4bfbc166-3dbf-4aae-96f5-515f59917846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 00:23:02.869936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:02.866637 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 00:23:02.869936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:02.866709 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" podUID="ed3d03e4-5550-40bc-8c00-48c3c7927fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 00:23:03.002218 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:03.002179 5001 patch_prober.go:29] interesting pod/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 00:23:03.002411 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:03.002243 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" podUID="a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 00:23:03.154996 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:03.153013 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:23:03.452993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:03.449085 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 00:23:03.452993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:03.449161 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" podUID="4bfbc166-3dbf-4aae-96f5-515f59917846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 00:23:03.872936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:03.872368 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 00:23:03.872936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:03.872433 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" podUID="ed3d03e4-5550-40bc-8c00-48c3c7927fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 00:23:04.003040 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:04.003000 5001 patch_prober.go:29] interesting pod/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 00:23:04.003243 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:04.003060 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" podUID="a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 00:23:04.434937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:04.434343 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 00:23:04.434937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:04.434410 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" podUID="4bfbc166-3dbf-4aae-96f5-515f59917846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 00:23:04.859628 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:04.859595 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-58dbc8bd4c-44p8c"] Oct 30 00:23:04.864143 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:04.860229 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 00:23:04.864368 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:04.864341 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" podUID="ed3d03e4-5550-40bc-8c00-48c3c7927fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 00:23:04.879949 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:04.879867 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-58dbc8bd4c-44p8c" Oct 30 00:23:04.920087 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:04.920051 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f52ea1-dockercfg-lzq8s\"" Oct 30 00:23:04.999100 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:04.961519 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f52ea1\"" Oct 30 00:23:05.013001 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:05.009820 5001 patch_prober.go:29] interesting pod/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 00:23:05.013001 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:05.009885 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" podUID="a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 00:23:05.029935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:05.024861 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4253945-fb6f-438c-937e-b5061e33c84b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f52ea1-58dbc8bd4c-44p8c\" (UID: \"a4253945-fb6f-438c-937e-b5061e33c84b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-58dbc8bd4c-44p8c" Oct 30 00:23:05.077936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:05.071573 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6nj8\" (UniqueName: \"kubernetes.io/projected/a4253945-fb6f-438c-937e-b5061e33c84b-kube-api-access-w6nj8\") pod \"payload-pre-processing-e2e-ipp-a-f52ea1-58dbc8bd4c-44p8c\" (UID: \"a4253945-fb6f-438c-937e-b5061e33c84b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-58dbc8bd4c-44p8c" Oct 30 00:23:05.178570 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:05.178465 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6nj8\" (UniqueName: \"kubernetes.io/projected/a4253945-fb6f-438c-937e-b5061e33c84b-kube-api-access-w6nj8\") pod \"payload-pre-processing-e2e-ipp-a-f52ea1-58dbc8bd4c-44p8c\" (UID: \"a4253945-fb6f-438c-937e-b5061e33c84b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-58dbc8bd4c-44p8c" Oct 30 00:23:05.179305 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:05.179273 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4253945-fb6f-438c-937e-b5061e33c84b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f52ea1-58dbc8bd4c-44p8c\" (UID: \"a4253945-fb6f-438c-937e-b5061e33c84b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-58dbc8bd4c-44p8c" Oct 30 00:23:05.180555 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:05.180529 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4253945-fb6f-438c-937e-b5061e33c84b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f52ea1-58dbc8bd4c-44p8c\" (UID: \"a4253945-fb6f-438c-937e-b5061e33c84b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-58dbc8bd4c-44p8c" Oct 30 00:23:05.189761 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:05.189734 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-7dcd79d5b6-ssgxn"] Oct 30 00:23:05.222272 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:05.222238 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-7dcd79d5b6-ssgxn" Oct 30 00:23:05.258727 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:05.258689 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6nj8\" (UniqueName: \"kubernetes.io/projected/a4253945-fb6f-438c-937e-b5061e33c84b-kube-api-access-w6nj8\") pod \"payload-pre-processing-e2e-ipp-a-f52ea1-58dbc8bd4c-44p8c\" (UID: \"a4253945-fb6f-438c-937e-b5061e33c84b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-58dbc8bd4c-44p8c" Oct 30 00:23:05.283935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:05.281967 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373a854c-4772-4632-b9fe-813747f39ca9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f52ea1-7dcd79d5b6-ssgxn\" (UID: \"373a854c-4772-4632-b9fe-813747f39ca9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-7dcd79d5b6-ssgxn" Oct 30 00:23:05.283935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:05.282183 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c48wk\" (UniqueName: \"kubernetes.io/projected/373a854c-4772-4632-b9fe-813747f39ca9-kube-api-access-c48wk\") pod \"payload-processing-e2e-ipp-a-f52ea1-7dcd79d5b6-ssgxn\" (UID: \"373a854c-4772-4632-b9fe-813747f39ca9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-7dcd79d5b6-ssgxn" Oct 30 00:23:05.337046 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:05.337007 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-58dbc8bd4c-44p8c" Oct 30 00:23:05.421947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:05.384846 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c48wk\" (UniqueName: \"kubernetes.io/projected/373a854c-4772-4632-b9fe-813747f39ca9-kube-api-access-c48wk\") pod \"payload-processing-e2e-ipp-a-f52ea1-7dcd79d5b6-ssgxn\" (UID: \"373a854c-4772-4632-b9fe-813747f39ca9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-7dcd79d5b6-ssgxn" Oct 30 00:23:05.421947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:05.409300 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373a854c-4772-4632-b9fe-813747f39ca9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f52ea1-7dcd79d5b6-ssgxn\" (UID: \"373a854c-4772-4632-b9fe-813747f39ca9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-7dcd79d5b6-ssgxn" Oct 30 00:23:05.421947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:05.410441 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373a854c-4772-4632-b9fe-813747f39ca9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f52ea1-7dcd79d5b6-ssgxn\" (UID: \"373a854c-4772-4632-b9fe-813747f39ca9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-7dcd79d5b6-ssgxn" Oct 30 00:23:05.454485 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:05.454380 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c48wk\" (UniqueName: \"kubernetes.io/projected/373a854c-4772-4632-b9fe-813747f39ca9-kube-api-access-c48wk\") pod \"payload-processing-e2e-ipp-a-f52ea1-7dcd79d5b6-ssgxn\" (UID: \"373a854c-4772-4632-b9fe-813747f39ca9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-7dcd79d5b6-ssgxn" Oct 30 00:23:05.654941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:05.632223 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-7dcd79d5b6-ssgxn" Oct 30 00:23:05.656001 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:05.645307 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:23:05.671149 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:05.671100 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:23:05.718491 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:05.718410 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:23:05.887030 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:05.886094 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 00:23:05.887030 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:05.886158 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" podUID="ed3d03e4-5550-40bc-8c00-48c3c7927fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 00:23:06.002652 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:06.002556 5001 patch_prober.go:29] interesting pod/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 00:23:06.002652 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:06.002616 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" podUID="a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 00:23:06.199039 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:06.199004 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-7dcd79d5b6-ssgxn"] Oct 30 00:23:06.206644 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:06.206567 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e2a011-tw9s7"] Oct 30 00:23:06.303038 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:06.303000 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7417dc-gncfw"] Oct 30 00:23:06.377569 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:06.377496 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-58dbc8bd4c-44p8c"] Oct 30 00:23:06.386879 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:06.386851 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35b82218-88cpk"] Oct 30 00:23:06.396692 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:06.396663 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-461609-g52q5"] Oct 30 00:23:06.937640 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:06.937604 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-7dcd79d5b6-ssgxn"] Oct 30 00:23:06.938145 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:06.937676 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-58dbc8bd4c-44p8c"] Oct 30 00:23:06.957816 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:23:06.957032 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4253945_fb6f_438c_937e_b5061e33c84b.slice/crio-585795ab829901416c950fb44c1bb364d5adacfe3654f80e126e90faa6c456a9 WatchSource:0}: Error finding container 585795ab829901416c950fb44c1bb364d5adacfe3654f80e126e90faa6c456a9: Status 404 returned error can't find the container with id 585795ab829901416c950fb44c1bb364d5adacfe3654f80e126e90faa6c456a9 Oct 30 00:23:06.959190 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:23:06.959150 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373a854c_4772_4632_b9fe_813747f39ca9.slice/crio-49c8be211b13cc96d59c7dd3bc46e7ece2e6a3e5026810a9c944ea1f0509f635 WatchSource:0}: Error finding container 49c8be211b13cc96d59c7dd3bc46e7ece2e6a3e5026810a9c944ea1f0509f635: Status 404 returned error can't find the container with id 49c8be211b13cc96d59c7dd3bc46e7ece2e6a3e5026810a9c944ea1f0509f635 Oct 30 00:23:06.987245 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:23:06.983603 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c83212_4e77_49a1_a39c_180047d688c0.slice/crio-f2bb4d4811725c27f0baa8f3ce62a89fb85d41c249855c79f22eff338455d494 WatchSource:0}: Error finding container f2bb4d4811725c27f0baa8f3ce62a89fb85d41c249855c79f22eff338455d494: Status 404 returned error can't find the container with id f2bb4d4811725c27f0baa8f3ce62a89fb85d41c249855c79f22eff338455d494 Oct 30 00:23:06.987245 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:06.986508 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e2a011-tw9s7"] Oct 30 00:23:07.008316 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:07.005881 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-461609-g52q5"] Oct 30 00:23:07.010764 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:07.010740 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:23:07.013051 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:07.013027 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:23:07.013624 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:07.013602 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7417dc-gncfw"] Oct 30 00:23:07.013873 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:07.013857 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:23:07.042217 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:07.042180 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35b82218-88cpk"] Oct 30 00:23:07.083537 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:07.083505 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:23:07.104385 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:07.104355 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:23:07.696782 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:07.696739 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-58dbc8bd4c-44p8c" event={"ID":"a4253945-fb6f-438c-937e-b5061e33c84b","Type":"ContainerStarted","Data":"585795ab829901416c950fb44c1bb364d5adacfe3654f80e126e90faa6c456a9"} Oct 30 00:23:07.698754 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:07.698722 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e2a011-tw9s7" event={"ID":"b8c83212-4e77-49a1-a39c-180047d688c0","Type":"ContainerStarted","Data":"f2bb4d4811725c27f0baa8f3ce62a89fb85d41c249855c79f22eff338455d494"} Oct 30 00:23:07.700980 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:07.700950 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b82218-88cpk" event={"ID":"f086c625-6fda-4000-9abf-1055f720d140","Type":"ContainerStarted","Data":"2f86e372e068379d6d30bba3b9a3c906826ee37cb22efbde099d25191c4ed247"} Oct 30 00:23:07.706237 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:07.703357 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-461609-g52q5" event={"ID":"cf5f5d9e-2d6f-4861-9279-ce514f89dd86","Type":"ContainerStarted","Data":"311921e76267bc51d89d1d70bb17453e094a1b9aec3eaf0f69508cbe17992fa7"} Oct 30 00:23:07.706237 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:07.705002 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-7dcd79d5b6-ssgxn" event={"ID":"373a854c-4772-4632-b9fe-813747f39ca9","Type":"ContainerStarted","Data":"49c8be211b13cc96d59c7dd3bc46e7ece2e6a3e5026810a9c944ea1f0509f635"} Oct 30 00:23:07.707121 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:07.707050 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7417dc-gncfw" event={"ID":"08ca0c60-e7de-43f7-8cac-db4372fa30d2","Type":"ContainerStarted","Data":"82c3f462263dc220224d2a0e39a46b3da39b21c2e47b55e3d7369e7649c0a89a"} Oct 30 00:23:08.850850 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:08.850809 5001 generic.go:356] "Generic (PLEG): container finished" podID="b8c83212-4e77-49a1-a39c-180047d688c0" containerID="c7495ab4a281a4d6591fbd11a8881bde1fb823476af06068a82a492871c8a673" exitCode=0 Oct 30 00:23:08.851330 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:08.850927 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e2a011-tw9s7" event={"ID":"b8c83212-4e77-49a1-a39c-180047d688c0","Type":"ContainerDied","Data":"c7495ab4a281a4d6591fbd11a8881bde1fb823476af06068a82a492871c8a673"} Oct 30 00:23:08.854003 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:08.853542 5001 generic.go:356] "Generic (PLEG): container finished" podID="f086c625-6fda-4000-9abf-1055f720d140" containerID="b55086178e9df08afa9b5f3f8f9bcc7064cfe69780845e5e5ba5b6d644613f4e" exitCode=0 Oct 30 00:23:08.854003 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:08.853628 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b82218-88cpk" event={"ID":"f086c625-6fda-4000-9abf-1055f720d140","Type":"ContainerDied","Data":"b55086178e9df08afa9b5f3f8f9bcc7064cfe69780845e5e5ba5b6d644613f4e"} Oct 30 00:23:08.856222 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:08.856197 5001 generic.go:356] "Generic (PLEG): container finished" podID="cf5f5d9e-2d6f-4861-9279-ce514f89dd86" containerID="8c979b1c5d79f2438c6cb7eebc6e1fd200d566c98770fbba03330199a07f18f0" exitCode=0 Oct 30 00:23:08.856334 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:08.856302 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-461609-g52q5" event={"ID":"cf5f5d9e-2d6f-4861-9279-ce514f89dd86","Type":"ContainerDied","Data":"8c979b1c5d79f2438c6cb7eebc6e1fd200d566c98770fbba03330199a07f18f0"} Oct 30 00:23:08.859755 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:08.859729 5001 generic.go:356] "Generic (PLEG): container finished" podID="08ca0c60-e7de-43f7-8cac-db4372fa30d2" containerID="ecbcb624b7cb9386becb98fdaa8765a7651b4ebd4ed2758d290f9093295342d0" exitCode=0 Oct 30 00:23:08.859949 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:08.859882 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7417dc-gncfw" event={"ID":"08ca0c60-e7de-43f7-8cac-db4372fa30d2","Type":"ContainerDied","Data":"ecbcb624b7cb9386becb98fdaa8765a7651b4ebd4ed2758d290f9093295342d0"} Oct 30 00:23:09.873323 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:09.873279 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-7dcd79d5b6-ssgxn" event={"ID":"373a854c-4772-4632-b9fe-813747f39ca9","Type":"ContainerStarted","Data":"81c802c8474f3d3cf0066d52bb9c8b9ececb180e37934f2920c9ed41fc00a5f3"} Oct 30 00:23:09.874372 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:09.873381 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-7dcd79d5b6-ssgxn" Oct 30 00:23:09.875353 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:09.875319 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-58dbc8bd4c-44p8c" event={"ID":"a4253945-fb6f-438c-937e-b5061e33c84b","Type":"ContainerStarted","Data":"388d571c14353142344f986aef92bac36806e0e1aa7e04d320bd6d21412d9a1b"} Oct 30 00:23:09.875465 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:09.875401 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-58dbc8bd4c-44p8c" Oct 30 00:23:09.922970 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:09.913506 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-7dcd79d5b6-ssgxn" podStartSLOduration=3.6705096 podStartE2EDuration="5.913486123s" totalImagesPullingTime="2.242976523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:23:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:23:09.909442634 +0000 UTC m=+1105.114943815" watchObservedRunningTime="2026-10-30 00:23:09.913486123 +0000 UTC m=+1105.118987305" Oct 30 00:23:09.964928 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:09.962139 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-58dbc8bd4c-44p8c" podStartSLOduration=3.689681645 podStartE2EDuration="5.962117665s" totalImagesPullingTime="2.27243602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:23:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:23:09.950146967 +0000 UTC m=+1105.155648149" watchObservedRunningTime="2026-10-30 00:23:09.962117665 +0000 UTC m=+1105.167618847" Oct 30 00:23:10.737765 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.604151 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-5dc97f76d7-9b4ng"] Oct 30 00:23:10.737765 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.611169 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-5dc97f76d7-9b4ng" Oct 30 00:23:10.737765 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.640321 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c34304-dockercfg-bdwcc\"" Oct 30 00:23:10.737765 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.691069 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-461609-g52q5" Oct 30 00:23:10.737765 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.710509 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqpj5\" (UniqueName: \"kubernetes.io/projected/bff05da7-3980-4634-be2f-6c3e1b160ec3-kube-api-access-nqpj5\") pod \"payload-pre-processing-e2e-ipp-a-c34304-5dc97f76d7-9b4ng\" (UID: \"bff05da7-3980-4634-be2f-6c3e1b160ec3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-5dc97f76d7-9b4ng" Oct 30 00:23:10.737765 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.710642 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bff05da7-3980-4634-be2f-6c3e1b160ec3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c34304-5dc97f76d7-9b4ng\" (UID: \"bff05da7-3980-4634-be2f-6c3e1b160ec3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-5dc97f76d7-9b4ng" Oct 30 00:23:10.737765 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.727953 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-5dc97f76d7-9b4ng"] Oct 30 00:23:10.740250 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.740006 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c34304-7ff4d749d8-fcpb6"] Oct 30 00:23:10.742840 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.742781 5001 cpu_manager.go:414] "containerMap: removing container" podUID="cf5f5d9e-2d6f-4861-9279-ce514f89dd86" containerName="revoke-keys" Oct 30 00:23:10.743031 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.743011 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf5f5d9e-2d6f-4861-9279-ce514f89dd86" containerName="revoke-keys" Oct 30 00:23:10.743467 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.743448 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf5f5d9e-2d6f-4861-9279-ce514f89dd86" containerName="revoke-keys" Oct 30 00:23:10.748540 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.747581 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c34304\"" Oct 30 00:23:10.752024 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.750797 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-7ff4d749d8-fcpb6" Oct 30 00:23:10.805142 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.805111 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c34304-7ff4d749d8-fcpb6"] Oct 30 00:23:10.813452 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.813418 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf5f5d9e-2d6f-4861-9279-ce514f89dd86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf5f5d9e-2d6f-4861-9279-ce514f89dd86-maas-api-xks-ca\") pod \"cf5f5d9e-2d6f-4861-9279-ce514f89dd86\" (UID: \"cf5f5d9e-2d6f-4861-9279-ce514f89dd86\") " Oct 30 00:23:10.816332 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.816302 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf5f5d9e-2d6f-4861-9279-ce514f89dd86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf5f5d9e-2d6f-4861-9279-ce514f89dd86-maas-api-openshift-ca\") pod \"cf5f5d9e-2d6f-4861-9279-ce514f89dd86\" (UID: \"cf5f5d9e-2d6f-4861-9279-ce514f89dd86\") " Oct 30 00:23:10.816456 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.816375 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf5f5d9e-2d6f-4861-9279-ce514f89dd86-kube-api-access-m5r4c\" (UniqueName: \"kubernetes.io/projected/cf5f5d9e-2d6f-4861-9279-ce514f89dd86-kube-api-access-m5r4c\") pod \"cf5f5d9e-2d6f-4861-9279-ce514f89dd86\" (UID: \"cf5f5d9e-2d6f-4861-9279-ce514f89dd86\") " Oct 30 00:23:10.817144 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.817116 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqpj5\" (UniqueName: \"kubernetes.io/projected/bff05da7-3980-4634-be2f-6c3e1b160ec3-kube-api-access-nqpj5\") pod \"payload-pre-processing-e2e-ipp-a-c34304-5dc97f76d7-9b4ng\" (UID: \"bff05da7-3980-4634-be2f-6c3e1b160ec3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-5dc97f76d7-9b4ng" Oct 30 00:23:10.817299 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.817284 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bff05da7-3980-4634-be2f-6c3e1b160ec3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c34304-5dc97f76d7-9b4ng\" (UID: \"bff05da7-3980-4634-be2f-6c3e1b160ec3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-5dc97f76d7-9b4ng" Oct 30 00:23:10.818298 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.818273 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bff05da7-3980-4634-be2f-6c3e1b160ec3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c34304-5dc97f76d7-9b4ng\" (UID: \"bff05da7-3980-4634-be2f-6c3e1b160ec3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-5dc97f76d7-9b4ng" Oct 30 00:23:10.818409 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.818309 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf5f5d9e-2d6f-4861-9279-ce514f89dd86-maas-api-openshift-ca" pod "cf5f5d9e-2d6f-4861-9279-ce514f89dd86" (UID: "cf5f5d9e-2d6f-4861-9279-ce514f89dd86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:23:10.822136 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.822097 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf5f5d9e-2d6f-4861-9279-ce514f89dd86-kube-api-access-m5r4c" pod "cf5f5d9e-2d6f-4861-9279-ce514f89dd86" (UID: "cf5f5d9e-2d6f-4861-9279-ce514f89dd86"). InnerVolumeSpecName "kube-api-access-m5r4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:23:10.834329 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.834304 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf5f5d9e-2d6f-4861-9279-ce514f89dd86-maas-api-xks-ca" pod "cf5f5d9e-2d6f-4861-9279-ce514f89dd86" (UID: "cf5f5d9e-2d6f-4861-9279-ce514f89dd86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:23:10.843106 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.841363 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e2a011-tw9s7" Oct 30 00:23:10.843106 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.841690 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqpj5\" (UniqueName: \"kubernetes.io/projected/bff05da7-3980-4634-be2f-6c3e1b160ec3-kube-api-access-nqpj5\") pod \"payload-pre-processing-e2e-ipp-a-c34304-5dc97f76d7-9b4ng\" (UID: \"bff05da7-3980-4634-be2f-6c3e1b160ec3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-5dc97f76d7-9b4ng" Oct 30 00:23:10.847539 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.847523 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b82218-88cpk" Oct 30 00:23:10.852643 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.852626 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7417dc-gncfw" Oct 30 00:23:10.888119 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.888089 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7417dc-gncfw" Oct 30 00:23:10.888539 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.888135 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7417dc-gncfw" event={"ID":"08ca0c60-e7de-43f7-8cac-db4372fa30d2","Type":"ContainerDied","Data":"82c3f462263dc220224d2a0e39a46b3da39b21c2e47b55e3d7369e7649c0a89a"} Oct 30 00:23:10.888539 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.888184 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82c3f462263dc220224d2a0e39a46b3da39b21c2e47b55e3d7369e7649c0a89a" Oct 30 00:23:10.889985 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.889961 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e2a011-tw9s7" event={"ID":"b8c83212-4e77-49a1-a39c-180047d688c0","Type":"ContainerDied","Data":"f2bb4d4811725c27f0baa8f3ce62a89fb85d41c249855c79f22eff338455d494"} Oct 30 00:23:10.890123 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.890005 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2bb4d4811725c27f0baa8f3ce62a89fb85d41c249855c79f22eff338455d494" Oct 30 00:23:10.890123 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.889967 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e2a011-tw9s7" Oct 30 00:23:10.891514 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.891496 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b82218-88cpk" Oct 30 00:23:10.891514 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.891509 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b82218-88cpk" event={"ID":"f086c625-6fda-4000-9abf-1055f720d140","Type":"ContainerDied","Data":"2f86e372e068379d6d30bba3b9a3c906826ee37cb22efbde099d25191c4ed247"} Oct 30 00:23:10.891696 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.891541 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f86e372e068379d6d30bba3b9a3c906826ee37cb22efbde099d25191c4ed247" Oct 30 00:23:10.893147 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.893132 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-461609-g52q5" Oct 30 00:23:10.893240 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.893166 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-461609-g52q5" event={"ID":"cf5f5d9e-2d6f-4861-9279-ce514f89dd86","Type":"ContainerDied","Data":"311921e76267bc51d89d1d70bb17453e094a1b9aec3eaf0f69508cbe17992fa7"} Oct 30 00:23:10.893240 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.893205 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="311921e76267bc51d89d1d70bb17453e094a1b9aec3eaf0f69508cbe17992fa7" Oct 30 00:23:10.927972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.922543 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e1c913-e273-48ef-a8e0-4903c1b32b30-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c34304-7ff4d749d8-fcpb6\" (UID: \"51e1c913-e273-48ef-a8e0-4903c1b32b30\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-7ff4d749d8-fcpb6" Oct 30 00:23:10.927972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.922618 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5zwr\" (UniqueName: \"kubernetes.io/projected/51e1c913-e273-48ef-a8e0-4903c1b32b30-kube-api-access-p5zwr\") pod \"payload-processing-e2e-ipp-a-c34304-7ff4d749d8-fcpb6\" (UID: \"51e1c913-e273-48ef-a8e0-4903c1b32b30\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-7ff4d749d8-fcpb6" Oct 30 00:23:10.927972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.923331 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf5f5d9e-2d6f-4861-9279-ce514f89dd86-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:23:10.927972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.923355 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5r4c\" (UniqueName: \"kubernetes.io/projected/cf5f5d9e-2d6f-4861-9279-ce514f89dd86-kube-api-access-m5r4c\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:23:10.927972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:10.923372 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf5f5d9e-2d6f-4861-9279-ce514f89dd86-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:23:11.023977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.023945 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08ca0c60-e7de-43f7-8cac-db4372fa30d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ca0c60-e7de-43f7-8cac-db4372fa30d2-maas-api-openshift-ca\") pod \"08ca0c60-e7de-43f7-8cac-db4372fa30d2\" (UID: \"08ca0c60-e7de-43f7-8cac-db4372fa30d2\") " Oct 30 00:23:11.024173 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.024009 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f086c625-6fda-4000-9abf-1055f720d140-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f086c625-6fda-4000-9abf-1055f720d140-maas-api-openshift-ca\") pod \"f086c625-6fda-4000-9abf-1055f720d140\" (UID: \"f086c625-6fda-4000-9abf-1055f720d140\") " Oct 30 00:23:11.024173 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.024114 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8c83212-4e77-49a1-a39c-180047d688c0-kube-api-access-62zwx\" (UniqueName: \"kubernetes.io/projected/b8c83212-4e77-49a1-a39c-180047d688c0-kube-api-access-62zwx\") pod \"b8c83212-4e77-49a1-a39c-180047d688c0\" (UID: \"b8c83212-4e77-49a1-a39c-180047d688c0\") " Oct 30 00:23:11.024306 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.024276 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ca0c60-e7de-43f7-8cac-db4372fa30d2-kube-api-access-4zwww\" (UniqueName: \"kubernetes.io/projected/08ca0c60-e7de-43f7-8cac-db4372fa30d2-kube-api-access-4zwww\") pod \"08ca0c60-e7de-43f7-8cac-db4372fa30d2\" (UID: \"08ca0c60-e7de-43f7-8cac-db4372fa30d2\") " Oct 30 00:23:11.024372 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.024354 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f086c625-6fda-4000-9abf-1055f720d140-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f086c625-6fda-4000-9abf-1055f720d140-maas-api-xks-ca\") pod \"f086c625-6fda-4000-9abf-1055f720d140\" (UID: \"f086c625-6fda-4000-9abf-1055f720d140\") " Oct 30 00:23:11.024497 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.024353 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08ca0c60-e7de-43f7-8cac-db4372fa30d2-maas-api-openshift-ca" pod "08ca0c60-e7de-43f7-8cac-db4372fa30d2" (UID: "08ca0c60-e7de-43f7-8cac-db4372fa30d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:23:11.024497 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.024407 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f086c625-6fda-4000-9abf-1055f720d140-maas-api-openshift-ca" pod "f086c625-6fda-4000-9abf-1055f720d140" (UID: "f086c625-6fda-4000-9abf-1055f720d140"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:23:11.024497 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.024430 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8c83212-4e77-49a1-a39c-180047d688c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c83212-4e77-49a1-a39c-180047d688c0-maas-api-openshift-ca\") pod \"b8c83212-4e77-49a1-a39c-180047d688c0\" (UID: \"b8c83212-4e77-49a1-a39c-180047d688c0\") " Oct 30 00:23:11.024696 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.024679 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08ca0c60-e7de-43f7-8cac-db4372fa30d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ca0c60-e7de-43f7-8cac-db4372fa30d2-maas-api-xks-ca\") pod \"08ca0c60-e7de-43f7-8cac-db4372fa30d2\" (UID: \"08ca0c60-e7de-43f7-8cac-db4372fa30d2\") " Oct 30 00:23:11.024778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.024751 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c83212-4e77-49a1-a39c-180047d688c0-maas-api-openshift-ca" pod "b8c83212-4e77-49a1-a39c-180047d688c0" (UID: "b8c83212-4e77-49a1-a39c-180047d688c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:23:11.024778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.024759 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8c83212-4e77-49a1-a39c-180047d688c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c83212-4e77-49a1-a39c-180047d688c0-maas-api-xks-ca\") pod \"b8c83212-4e77-49a1-a39c-180047d688c0\" (UID: \"b8c83212-4e77-49a1-a39c-180047d688c0\") " Oct 30 00:23:11.024928 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.024871 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f086c625-6fda-4000-9abf-1055f720d140-kube-api-access-fdrpw\" (UniqueName: \"kubernetes.io/projected/f086c625-6fda-4000-9abf-1055f720d140-kube-api-access-fdrpw\") pod \"f086c625-6fda-4000-9abf-1055f720d140\" (UID: \"f086c625-6fda-4000-9abf-1055f720d140\") " Oct 30 00:23:11.025655 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.025626 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e1c913-e273-48ef-a8e0-4903c1b32b30-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c34304-7ff4d749d8-fcpb6\" (UID: \"51e1c913-e273-48ef-a8e0-4903c1b32b30\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-7ff4d749d8-fcpb6" Oct 30 00:23:11.025813 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.025703 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5zwr\" (UniqueName: \"kubernetes.io/projected/51e1c913-e273-48ef-a8e0-4903c1b32b30-kube-api-access-p5zwr\") pod \"payload-processing-e2e-ipp-a-c34304-7ff4d749d8-fcpb6\" (UID: \"51e1c913-e273-48ef-a8e0-4903c1b32b30\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-7ff4d749d8-fcpb6" Oct 30 00:23:11.028346 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.025980 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c83212-4e77-49a1-a39c-180047d688c0-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:23:11.028346 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.026014 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ca0c60-e7de-43f7-8cac-db4372fa30d2-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:23:11.028346 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.026031 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f086c625-6fda-4000-9abf-1055f720d140-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:23:11.061249 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.044859 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e1c913-e273-48ef-a8e0-4903c1b32b30-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c34304-7ff4d749d8-fcpb6\" (UID: \"51e1c913-e273-48ef-a8e0-4903c1b32b30\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-7ff4d749d8-fcpb6" Oct 30 00:23:11.061249 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.054133 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f086c625-6fda-4000-9abf-1055f720d140-maas-api-xks-ca" pod "f086c625-6fda-4000-9abf-1055f720d140" (UID: "f086c625-6fda-4000-9abf-1055f720d140"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:23:11.061249 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.054437 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ca0c60-e7de-43f7-8cac-db4372fa30d2-kube-api-access-4zwww" pod "08ca0c60-e7de-43f7-8cac-db4372fa30d2" (UID: "08ca0c60-e7de-43f7-8cac-db4372fa30d2"). InnerVolumeSpecName "kube-api-access-4zwww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:23:11.061249 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.057579 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-5dc97f76d7-9b4ng" Oct 30 00:23:11.061249 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.058850 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c83212-4e77-49a1-a39c-180047d688c0-kube-api-access-62zwx" pod "b8c83212-4e77-49a1-a39c-180047d688c0" (UID: "b8c83212-4e77-49a1-a39c-180047d688c0"). InnerVolumeSpecName "kube-api-access-62zwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:23:11.085555 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.085522 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08ca0c60-e7de-43f7-8cac-db4372fa30d2-maas-api-xks-ca" pod "08ca0c60-e7de-43f7-8cac-db4372fa30d2" (UID: "08ca0c60-e7de-43f7-8cac-db4372fa30d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:23:11.087282 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.087248 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f086c625-6fda-4000-9abf-1055f720d140-kube-api-access-fdrpw" pod "f086c625-6fda-4000-9abf-1055f720d140" (UID: "f086c625-6fda-4000-9abf-1055f720d140"). InnerVolumeSpecName "kube-api-access-fdrpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:23:11.092800 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.092347 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8c83212-4e77-49a1-a39c-180047d688c0-maas-api-xks-ca" pod "b8c83212-4e77-49a1-a39c-180047d688c0" (UID: "b8c83212-4e77-49a1-a39c-180047d688c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:23:11.103962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.094906 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5zwr\" (UniqueName: \"kubernetes.io/projected/51e1c913-e273-48ef-a8e0-4903c1b32b30-kube-api-access-p5zwr\") pod \"payload-processing-e2e-ipp-a-c34304-7ff4d749d8-fcpb6\" (UID: \"51e1c913-e273-48ef-a8e0-4903c1b32b30\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-7ff4d749d8-fcpb6" Oct 30 00:23:11.140124 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.139710 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ca0c60-e7de-43f7-8cac-db4372fa30d2-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:23:11.140124 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.139752 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c83212-4e77-49a1-a39c-180047d688c0-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:23:11.140124 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.139766 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdrpw\" (UniqueName: \"kubernetes.io/projected/f086c625-6fda-4000-9abf-1055f720d140-kube-api-access-fdrpw\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:23:11.140124 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.139785 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62zwx\" (UniqueName: \"kubernetes.io/projected/b8c83212-4e77-49a1-a39c-180047d688c0-kube-api-access-62zwx\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:23:11.140124 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.139798 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zwww\" (UniqueName: \"kubernetes.io/projected/08ca0c60-e7de-43f7-8cac-db4372fa30d2-kube-api-access-4zwww\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:23:11.140124 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.139811 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f086c625-6fda-4000-9abf-1055f720d140-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:23:11.141718 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.141386 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-7ff4d749d8-fcpb6" Oct 30 00:23:11.505843 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:23:11.505817 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbff05da7_3980_4634_be2f_6c3e1b160ec3.slice/crio-b498075a08be50a5367b1399a7e565a421e25656a54fbd5d472a3922ffcf5822 WatchSource:0}: Error finding container b498075a08be50a5367b1399a7e565a421e25656a54fbd5d472a3922ffcf5822: Status 404 returned error can't find the container with id b498075a08be50a5367b1399a7e565a421e25656a54fbd5d472a3922ffcf5822 Oct 30 00:23:11.506948 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.506819 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-5dc97f76d7-9b4ng"] Oct 30 00:23:11.543237 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:23:11.543209 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e1c913_e273_48ef_a8e0_4903c1b32b30.slice/crio-72cc003670b42db9d29cd61cc044de430e2d842b5c4785f5cb2be5df27220de2 WatchSource:0}: Error finding container 72cc003670b42db9d29cd61cc044de430e2d842b5c4785f5cb2be5df27220de2: Status 404 returned error can't find the container with id 72cc003670b42db9d29cd61cc044de430e2d842b5c4785f5cb2be5df27220de2 Oct 30 00:23:11.544705 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.544687 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c34304-7ff4d749d8-fcpb6"] Oct 30 00:23:11.906803 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.906707 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-5dc97f76d7-9b4ng" event={"ID":"bff05da7-3980-4634-be2f-6c3e1b160ec3","Type":"ContainerStarted","Data":"40ba89477e646f467218b54a010d1df96dc3c520dc214c48e0b415f1580f3eeb"} Oct 30 00:23:11.906803 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.906770 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-5dc97f76d7-9b4ng" event={"ID":"bff05da7-3980-4634-be2f-6c3e1b160ec3","Type":"ContainerStarted","Data":"b498075a08be50a5367b1399a7e565a421e25656a54fbd5d472a3922ffcf5822"} Oct 30 00:23:11.906803 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.906801 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-5dc97f76d7-9b4ng" Oct 30 00:23:11.909451 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.909425 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-7ff4d749d8-fcpb6" event={"ID":"51e1c913-e273-48ef-a8e0-4903c1b32b30","Type":"ContainerStarted","Data":"2cefe897084883582a0f63911b97b8a2a3cd74cb1461d6809f0fd8ebea848bb6"} Oct 30 00:23:11.909611 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.909500 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-7ff4d749d8-fcpb6" Oct 30 00:23:11.909611 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.909519 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-7ff4d749d8-fcpb6" event={"ID":"51e1c913-e273-48ef-a8e0-4903c1b32b30","Type":"ContainerStarted","Data":"72cc003670b42db9d29cd61cc044de430e2d842b5c4785f5cb2be5df27220de2"} Oct 30 00:23:11.953296 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.953238 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-5dc97f76d7-9b4ng" podStartSLOduration=1.953221402 podStartE2EDuration="1.953221402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:23:11.94653195 +0000 UTC m=+1107.152033130" watchObservedRunningTime="2026-10-30 00:23:11.953221402 +0000 UTC m=+1107.158722584" Oct 30 00:23:11.986715 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:11.986657 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-7ff4d749d8-fcpb6" podStartSLOduration=1.9866439150000001 podStartE2EDuration="1.986643915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:23:11.984300149 +0000 UTC m=+1107.189801329" watchObservedRunningTime="2026-10-30 00:23:11.986643915 +0000 UTC m=+1107.192145097" Oct 30 00:23:16.494930 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:16.493220 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7417dc-cdd68c66c-cjxww"] Oct 30 00:23:17.503932 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:23:17.500205 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:23:18.143040 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:18.143007 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e2a011-dd5bd7995-m4h7w"] Oct 30 00:23:20.895644 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:20.895613 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-58dbc8bd4c-44p8c" Oct 30 00:23:20.896261 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:20.895685 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-7dcd79d5b6-ssgxn" Oct 30 00:23:21.493629 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:21.493585 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7417dc-cdd68c66c-cjxww" podUID="c115ecb3-2006-4756-a8cf-d06a56ebd4f7" containerName="payload-pre-processing" containerID="cri-o://9630c2c6b484fac1bb9cf840dd489f9b75e039dc86adf4c61230a17606287bd1" gracePeriod=25 Oct 30 00:23:21.788359 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:21.788329 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7417dc-cdd68c66c-cjxww" Oct 30 00:23:21.862148 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:21.862114 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c115ecb3-2006-4756-a8cf-d06a56ebd4f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c115ecb3-2006-4756-a8cf-d06a56ebd4f7-plugins-config-volume\") pod \"c115ecb3-2006-4756-a8cf-d06a56ebd4f7\" (UID: \"c115ecb3-2006-4756-a8cf-d06a56ebd4f7\") " Oct 30 00:23:21.862335 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:21.862180 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c115ecb3-2006-4756-a8cf-d06a56ebd4f7-kube-api-access-j87bt\" (UniqueName: \"kubernetes.io/projected/c115ecb3-2006-4756-a8cf-d06a56ebd4f7-kube-api-access-j87bt\") pod \"c115ecb3-2006-4756-a8cf-d06a56ebd4f7\" (UID: \"c115ecb3-2006-4756-a8cf-d06a56ebd4f7\") " Oct 30 00:23:21.862508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:21.862486 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c115ecb3-2006-4756-a8cf-d06a56ebd4f7-plugins-config-volume" pod "c115ecb3-2006-4756-a8cf-d06a56ebd4f7" (UID: "c115ecb3-2006-4756-a8cf-d06a56ebd4f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:23:21.863009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:21.862986 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c115ecb3-2006-4756-a8cf-d06a56ebd4f7-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:23:21.864619 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:21.864583 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c115ecb3-2006-4756-a8cf-d06a56ebd4f7-kube-api-access-j87bt" pod "c115ecb3-2006-4756-a8cf-d06a56ebd4f7" (UID: "c115ecb3-2006-4756-a8cf-d06a56ebd4f7"). InnerVolumeSpecName "kube-api-access-j87bt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:23:21.981011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:21.975136 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j87bt\" (UniqueName: \"kubernetes.io/projected/c115ecb3-2006-4756-a8cf-d06a56ebd4f7-kube-api-access-j87bt\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:23:22.301635 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:22.301599 5001 generic.go:356] "Generic (PLEG): container finished" podID="c115ecb3-2006-4756-a8cf-d06a56ebd4f7" containerID="9630c2c6b484fac1bb9cf840dd489f9b75e039dc86adf4c61230a17606287bd1" exitCode=0 Oct 30 00:23:22.301813 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:22.301642 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7417dc-cdd68c66c-cjxww" Oct 30 00:23:22.301813 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:22.301651 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7417dc-cdd68c66c-cjxww" event={"ID":"c115ecb3-2006-4756-a8cf-d06a56ebd4f7","Type":"ContainerDied","Data":"9630c2c6b484fac1bb9cf840dd489f9b75e039dc86adf4c61230a17606287bd1"} Oct 30 00:23:22.301813 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:22.301710 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7417dc-cdd68c66c-cjxww" event={"ID":"c115ecb3-2006-4756-a8cf-d06a56ebd4f7","Type":"ContainerDied","Data":"8041d50accf15e086cef3cd2a02d1f26807ea6a4741602b85c9e3b532ce093e0"} Oct 30 00:23:22.301813 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:22.301735 5001 scope.go:122] "RemoveContainer" containerID="9630c2c6b484fac1bb9cf840dd489f9b75e039dc86adf4c61230a17606287bd1" Oct 30 00:23:22.313895 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:22.313877 5001 scope.go:122] "RemoveContainer" containerID="9630c2c6b484fac1bb9cf840dd489f9b75e039dc86adf4c61230a17606287bd1" Oct 30 00:23:22.314186 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:23:22.314163 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9630c2c6b484fac1bb9cf840dd489f9b75e039dc86adf4c61230a17606287bd1\": container with ID starting with 9630c2c6b484fac1bb9cf840dd489f9b75e039dc86adf4c61230a17606287bd1 not found: ID does not exist" containerID="9630c2c6b484fac1bb9cf840dd489f9b75e039dc86adf4c61230a17606287bd1" Oct 30 00:23:22.314236 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:22.314197 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9630c2c6b484fac1bb9cf840dd489f9b75e039dc86adf4c61230a17606287bd1"} err="failed to get container status \"9630c2c6b484fac1bb9cf840dd489f9b75e039dc86adf4c61230a17606287bd1\": rpc error: code = NotFound desc = could not find container \"9630c2c6b484fac1bb9cf840dd489f9b75e039dc86adf4c61230a17606287bd1\": container with ID starting with 9630c2c6b484fac1bb9cf840dd489f9b75e039dc86adf4c61230a17606287bd1 not found: ID does not exist" Oct 30 00:23:22.352089 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:22.352064 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7417dc-cdd68c66c-cjxww"] Oct 30 00:23:22.361288 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:22.361266 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7417dc-cdd68c66c-cjxww"] Oct 30 00:23:22.922258 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:22.922226 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c115ecb3-2006-4756-a8cf-d06a56ebd4f7" path="/var/lib/kubelet/pods/c115ecb3-2006-4756-a8cf-d06a56ebd4f7/volumes" Oct 30 00:23:22.927060 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:22.927037 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-7ff4d749d8-fcpb6" Oct 30 00:23:22.927199 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:22.927111 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-5dc97f76d7-9b4ng" Oct 30 00:23:23.143527 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:23.143465 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e2a011-dd5bd7995-m4h7w" podUID="1b3b2f8d-f573-4956-8b85-504d533c8fce" containerName="payload-pre-processing" containerID="cri-o://2cde9ae2c0804eb343ffc896baae6a7d08e6fd4efc49eae0e70a11a7305ac9dc" gracePeriod=25 Oct 30 00:23:23.334963 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:23.325789 5001 generic.go:356] "Generic (PLEG): container finished" podID="1b3b2f8d-f573-4956-8b85-504d533c8fce" containerID="2cde9ae2c0804eb343ffc896baae6a7d08e6fd4efc49eae0e70a11a7305ac9dc" exitCode=0 Oct 30 00:23:23.334963 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:23.326153 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e2a011-dd5bd7995-m4h7w" event={"ID":"1b3b2f8d-f573-4956-8b85-504d533c8fce","Type":"ContainerDied","Data":"2cde9ae2c0804eb343ffc896baae6a7d08e6fd4efc49eae0e70a11a7305ac9dc"} Oct 30 00:23:23.567199 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:23.566825 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e2a011-dd5bd7995-m4h7w" Oct 30 00:23:23.641516 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:23.641482 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b3b2f8d-f573-4956-8b85-504d533c8fce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3b2f8d-f573-4956-8b85-504d533c8fce-plugins-config-volume\") pod \"1b3b2f8d-f573-4956-8b85-504d533c8fce\" (UID: \"1b3b2f8d-f573-4956-8b85-504d533c8fce\") " Oct 30 00:23:23.641695 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:23.641633 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b3b2f8d-f573-4956-8b85-504d533c8fce-kube-api-access-ll44h\" (UniqueName: \"kubernetes.io/projected/1b3b2f8d-f573-4956-8b85-504d533c8fce-kube-api-access-ll44h\") pod \"1b3b2f8d-f573-4956-8b85-504d533c8fce\" (UID: \"1b3b2f8d-f573-4956-8b85-504d533c8fce\") " Oct 30 00:23:23.641884 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:23.641860 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b3b2f8d-f573-4956-8b85-504d533c8fce-plugins-config-volume" pod "1b3b2f8d-f573-4956-8b85-504d533c8fce" (UID: "1b3b2f8d-f573-4956-8b85-504d533c8fce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:23:23.642470 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:23.642448 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3b2f8d-f573-4956-8b85-504d533c8fce-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:23:23.644253 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:23.644221 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b3b2f8d-f573-4956-8b85-504d533c8fce-kube-api-access-ll44h" pod "1b3b2f8d-f573-4956-8b85-504d533c8fce" (UID: "1b3b2f8d-f573-4956-8b85-504d533c8fce"). InnerVolumeSpecName "kube-api-access-ll44h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:23:23.744441 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:23.744394 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll44h\" (UniqueName: \"kubernetes.io/projected/1b3b2f8d-f573-4956-8b85-504d533c8fce-kube-api-access-ll44h\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:23:24.338210 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:24.338180 5001 generic.go:356] "Generic (PLEG): container finished" podID="3febb7a0-47fa-4545-b76c-977789498353" containerID="77fb49a7245c9250222fd1f7a88c1f0a80dc41aa4ce8e9b9ab9273f55a12bc84" exitCode=1 Oct 30 00:23:24.338666 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:24.338250 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4n6qq" event={"ID":"3febb7a0-47fa-4545-b76c-977789498353","Type":"ContainerDied","Data":"77fb49a7245c9250222fd1f7a88c1f0a80dc41aa4ce8e9b9ab9273f55a12bc84"} Oct 30 00:23:24.338737 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:24.338700 5001 scope.go:122] "RemoveContainer" containerID="77fb49a7245c9250222fd1f7a88c1f0a80dc41aa4ce8e9b9ab9273f55a12bc84" Oct 30 00:23:24.339921 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:24.339886 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e2a011-dd5bd7995-m4h7w" Oct 30 00:23:24.340042 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:24.339900 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e2a011-dd5bd7995-m4h7w" event={"ID":"1b3b2f8d-f573-4956-8b85-504d533c8fce","Type":"ContainerDied","Data":"38197857ab9e581400b1ac45dceb3ce0be2233b989cefe1ac27cde21c12a0997"} Oct 30 00:23:24.340042 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:24.339967 5001 scope.go:122] "RemoveContainer" containerID="2cde9ae2c0804eb343ffc896baae6a7d08e6fd4efc49eae0e70a11a7305ac9dc" Oct 30 00:23:24.422595 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:24.422563 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e2a011-dd5bd7995-m4h7w"] Oct 30 00:23:24.429056 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:24.429028 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e2a011-dd5bd7995-m4h7w"] Oct 30 00:23:25.023976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:25.023940 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b3b2f8d-f573-4956-8b85-504d533c8fce" path="/var/lib/kubelet/pods/1b3b2f8d-f573-4956-8b85-504d533c8fce/volumes" Oct 30 00:23:25.351746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:25.351655 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4n6qq" event={"ID":"3febb7a0-47fa-4545-b76c-977789498353","Type":"ContainerStarted","Data":"0123ac54213019c7f6cf2f02f731454f8f91ed49855edecadc9f893af1bafba8"} Oct 30 00:23:25.352211 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:25.351876 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4n6qq" Oct 30 00:23:26.903599 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:26.903514 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg"] Oct 30 00:23:26.904787 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:26.904763 5001 cpu_manager.go:414] "containerMap: removing container" podUID="c115ecb3-2006-4756-a8cf-d06a56ebd4f7" containerName="payload-pre-processing" Oct 30 00:23:26.905050 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:26.904952 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c115ecb3-2006-4756-a8cf-d06a56ebd4f7" containerName="payload-pre-processing" Oct 30 00:23:26.905050 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:26.904987 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1b3b2f8d-f573-4956-8b85-504d533c8fce" containerName="payload-pre-processing" Oct 30 00:23:26.905050 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:26.904995 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b3b2f8d-f573-4956-8b85-504d533c8fce" containerName="payload-pre-processing" Oct 30 00:23:26.905050 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:26.905038 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b8c83212-4e77-49a1-a39c-180047d688c0" containerName="revoke-keys" Oct 30 00:23:26.905050 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:26.905046 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8c83212-4e77-49a1-a39c-180047d688c0" containerName="revoke-keys" Oct 30 00:23:26.905380 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:26.905066 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f086c625-6fda-4000-9abf-1055f720d140" containerName="revoke-keys" Oct 30 00:23:26.905380 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:26.905073 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f086c625-6fda-4000-9abf-1055f720d140" containerName="revoke-keys" Oct 30 00:23:26.905380 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:26.905089 5001 cpu_manager.go:414] "containerMap: removing container" podUID="08ca0c60-e7de-43f7-8cac-db4372fa30d2" containerName="revoke-keys" Oct 30 00:23:26.905380 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:26.905095 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ca0c60-e7de-43f7-8cac-db4372fa30d2" containerName="revoke-keys" Oct 30 00:23:26.905380 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:26.905324 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8c83212-4e77-49a1-a39c-180047d688c0" containerName="revoke-keys" Oct 30 00:23:26.905380 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:26.905341 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f086c625-6fda-4000-9abf-1055f720d140" containerName="revoke-keys" Oct 30 00:23:26.905380 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:26.905350 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="08ca0c60-e7de-43f7-8cac-db4372fa30d2" containerName="revoke-keys" Oct 30 00:23:26.905380 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:26.905373 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c115ecb3-2006-4756-a8cf-d06a56ebd4f7" containerName="payload-pre-processing" Oct 30 00:23:26.905380 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:26.905384 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b3b2f8d-f573-4956-8b85-504d533c8fce" containerName="payload-pre-processing" Oct 30 00:23:26.923974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:26.909606 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:23:26.927793 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:26.926097 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-96027e-openshift-default-dockercfg-t8qds\"" Oct 30 00:23:26.952039 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:26.952008 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg"] Oct 30 00:23:27.016932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:27.016745 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3868e018-e83d-4eb5-9aff-842760b30295-istiod-ca-cert\") pod \"e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:23:27.016932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:27.016864 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-credential-socket\") pod \"e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:23:27.017216 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:27.017012 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3868e018-e83d-4eb5-9aff-842760b30295-istio-podinfo\") pod \"e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:23:27.017216 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:27.017067 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-workload-socket\") pod \"e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:23:27.017347 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:27.017272 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-istio-data\") pod \"e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:23:27.017347 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:27.017317 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-istio-envoy\") pod \"e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:23:27.018982 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:27.017368 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-workload-certs\") pod \"e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:23:27.018982 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:27.018273 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnkn8\" (UniqueName: \"kubernetes.io/projected/3868e018-e83d-4eb5-9aff-842760b30295-kube-api-access-mnkn8\") pod \"e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:23:27.018982 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:27.018366 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3868e018-e83d-4eb5-9aff-842760b30295-istio-token\") pod \"e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:23:27.127450 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:27.127409 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3868e018-e83d-4eb5-9aff-842760b30295-istio-podinfo\") pod \"e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:23:27.144019 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:27.143984 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-workload-socket\") pod \"e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:23:27.159981 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:27.144421 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-istio-data\") pod \"e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:23:27.160306 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:27.160276 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-istio-envoy\") pod \"e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:23:27.160541 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:27.160513 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-workload-certs\") pod \"e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:23:27.160687 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:27.160668 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnkn8\" (UniqueName: \"kubernetes.io/projected/3868e018-e83d-4eb5-9aff-842760b30295-kube-api-access-mnkn8\") pod \"e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:23:27.160887 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:27.160871 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3868e018-e83d-4eb5-9aff-842760b30295-istio-token\") pod \"e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:23:27.161097 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:27.161075 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3868e018-e83d-4eb5-9aff-842760b30295-istiod-ca-cert\") pod \"e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:23:27.161304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:27.161288 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-credential-socket\") pod \"e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:23:27.161848 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:27.161826 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-credential-socket\") pod \"e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:23:27.161997 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:27.153433 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-workload-socket\") pod \"e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:23:27.164991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:27.154840 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-istio-data\") pod \"e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:23:27.166354 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:27.165703 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-workload-certs\") pod \"e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:23:27.166634 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:27.166293 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3868e018-e83d-4eb5-9aff-842760b30295-istiod-ca-cert\") pod \"e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:23:27.203428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:27.203394 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-istio-envoy\") pod \"e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:23:27.230937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:27.209467 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3868e018-e83d-4eb5-9aff-842760b30295-istio-podinfo\") pod \"e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:23:27.251935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:27.236433 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnkn8\" (UniqueName: \"kubernetes.io/projected/3868e018-e83d-4eb5-9aff-842760b30295-kube-api-access-mnkn8\") pod \"e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:23:27.253074 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:27.252993 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3868e018-e83d-4eb5-9aff-842760b30295-istio-token\") pod \"e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:23:27.550353 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:27.550316 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:23:28.839809 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:23:28.839777 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3868e018_e83d_4eb5_9aff_842760b30295.slice/crio-e4e4936facaa988859d5ca88458f9f24c7d19712b9f3421f6e6da385dfb85a08 WatchSource:0}: Error finding container e4e4936facaa988859d5ca88458f9f24c7d19712b9f3421f6e6da385dfb85a08: Status 404 returned error can't find the container with id e4e4936facaa988859d5ca88458f9f24c7d19712b9f3421f6e6da385dfb85a08 Oct 30 00:23:28.840276 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:28.839872 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg"] Oct 30 00:23:28.843149 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:28.843113 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:23:28.843274 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:28.843177 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:23:28.843274 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:28.843207 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:23:29.623582 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:29.623541 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" event={"ID":"3868e018-e83d-4eb5-9aff-842760b30295","Type":"ContainerStarted","Data":"a8e31ad8114053cd4b4a790aa3a7450c47ae87188ac54c5dce8602bab07e312b"} Oct 30 00:23:29.623770 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:29.623609 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" event={"ID":"3868e018-e83d-4eb5-9aff-842760b30295","Type":"ContainerStarted","Data":"e4e4936facaa988859d5ca88458f9f24c7d19712b9f3421f6e6da385dfb85a08"} Oct 30 00:23:30.551538 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:30.551499 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:23:30.557360 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:30.557333 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:23:30.611198 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:30.611146 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" podStartSLOduration=4.6111312269999996 podStartE2EDuration="4.611131227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:23:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:23:29.682148922 +0000 UTC m=+1124.887650103" watchObservedRunningTime="2026-10-30 00:23:30.611131227 +0000 UTC m=+1125.816632407" Oct 30 00:23:30.634943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:30.634899 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:23:30.635976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:30.635951 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:23:34.108448 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:34.108412 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-010356eb-zhrqb"] Oct 30 00:23:34.115347 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:34.115316 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-010356eb-zhrqb"] Oct 30 00:23:34.921425 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:34.921391 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8c92d1d-c016-49f5-a7ae-971c9702f505" path="/var/lib/kubelet/pods/a8c92d1d-c016-49f5-a7ae-971c9702f505/volumes" Oct 30 00:23:36.345052 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:36.345012 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-nxq9b"] Oct 30 00:23:36.492233 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:36.492195 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-nxq9b"] Oct 30 00:23:36.921567 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:36.921533 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22ff6405-8fac-477b-aee9-08e629f42d8c" path="/var/lib/kubelet/pods/22ff6405-8fac-477b-aee9-08e629f42d8c/volumes" Oct 30 00:23:45.031878 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:45.031846 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9c3d9cc-42dv5"] Oct 30 00:23:45.040658 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:45.040633 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9c3d9cc-42dv5"] Oct 30 00:23:46.368700 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:46.368664 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4n6qq" Oct 30 00:23:46.922233 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:46.922199 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb41aa3a-14f1-4213-af4b-80ced32a857a" path="/var/lib/kubelet/pods/fb41aa3a-14f1-4213-af4b-80ced32a857a/volumes" Oct 30 00:23:47.325204 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:23:47.325158 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:23:49.082006 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:49.081969 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9960d3c-fkjdx"] Oct 30 00:23:49.206824 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:49.206785 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9960d3c-fkjdx"] Oct 30 00:23:49.317299 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:49.317267 5001 scope.go:122] "RemoveContainer" containerID="4ba564bf722c3c02baf113ffffee65efe90fb664a3b44694419e1ce7855df504" Oct 30 00:23:49.446175 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:49.446146 5001 scope.go:122] "RemoveContainer" containerID="3c9847498b93bbcd85e141dfa00b00ff8d0c939c1b9272e3e90d1f664b0bae5a" Oct 30 00:23:50.158045 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:50.157963 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7417dc-gncfw"] Oct 30 00:23:50.188307 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:50.188274 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7417dc-gncfw"] Oct 30 00:23:50.921695 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:50.921666 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08ca0c60-e7de-43f7-8cac-db4372fa30d2" path="/var/lib/kubelet/pods/08ca0c60-e7de-43f7-8cac-db4372fa30d2/volumes" Oct 30 00:23:50.922278 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:50.922261 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5bf89d2-732e-4a71-8ba3-7db0affe4155" path="/var/lib/kubelet/pods/b5bf89d2-732e-4a71-8ba3-7db0affe4155/volumes" Oct 30 00:23:57.989187 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:57.989156 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8"] Oct 30 00:23:57.989679 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:57.989398 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" podUID="97f84a41-7ac2-4734-ab36-ebf29a62b8fa" containerName="istio-proxy" containerID="cri-o://d3be5d676efea82110af81195d4e7082d4262d35d85361fa99390aabeb519fbc" gracePeriod=30 Oct 30 00:23:59.030856 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:59.030825 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cf9bf0c-sc5xw"] Oct 30 00:23:59.040268 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:23:59.040231 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cf9bf0c-sc5xw"] Oct 30 00:24:00.600938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:00.583546 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e2a011-tw9s7"] Oct 30 00:24:00.811485 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:00.811449 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e2a011-tw9s7"] Oct 30 00:24:00.933179 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:00.933079 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:24:01.057523 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.057480 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:24:01.081985 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.076250 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:24:01.364817 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.364761 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0649729d-adb7-4ed6-90da-853e44f73236" path="/var/lib/kubelet/pods/0649729d-adb7-4ed6-90da-853e44f73236/volumes" Oct 30 00:24:01.366054 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.366027 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8c83212-4e77-49a1-a39c-180047d688c0" path="/var/lib/kubelet/pods/b8c83212-4e77-49a1-a39c-180047d688c0/volumes" Oct 30 00:24:01.366778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.366719 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l"] Oct 30 00:24:01.376270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.376243 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l"] Oct 30 00:24:01.376544 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.376473 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:24:01.399275 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.399248 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f8397b-openshift-default-dockercfg-8fqz9\"" Oct 30 00:24:01.468338 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.467556 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istio-token\") pod \"e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:24:01.468338 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.467643 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgg26\" (UniqueName: \"kubernetes.io/projected/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-kube-api-access-xgg26\") pod \"e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:24:01.468338 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.467702 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-credential-socket\") pod \"e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:24:01.468338 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.467736 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istio-podinfo\") pod \"e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:24:01.468338 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.467817 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-workload-socket\") pod \"e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:24:01.468338 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.467865 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istio-data\") pod \"e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:24:01.468338 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.467984 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istio-envoy\") pod \"e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:24:01.468338 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.468036 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-workload-certs\") pod \"e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:24:01.468338 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.468059 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istiod-ca-cert\") pod \"e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:24:01.594392 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.593868 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istio-token\") pod \"e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:24:01.628387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.594552 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgg26\" (UniqueName: \"kubernetes.io/projected/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-kube-api-access-xgg26\") pod \"e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:24:01.628387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.595009 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-credential-socket\") pod \"e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:24:01.628387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.596305 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-credential-socket\") pod \"e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:24:01.628387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.596421 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istio-podinfo\") pod \"e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:24:01.628387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.596541 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-workload-socket\") pod \"e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:24:01.628387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.597117 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istio-data\") pod \"e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:24:01.628387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.597378 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-workload-socket\") pod \"e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:24:01.628387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.596618 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istio-data\") pod \"e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:24:01.628387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.597580 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istio-envoy\") pod \"e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:24:01.628387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.597666 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-workload-certs\") pod \"e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:24:01.628387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.598245 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-workload-certs\") pod \"e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:24:01.628387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.598470 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istiod-ca-cert\") pod \"e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:24:01.628387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.599330 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istiod-ca-cert\") pod \"e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:24:01.628387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.603531 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istio-envoy\") pod \"e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:24:01.628387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.620408 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istio-token\") pod \"e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:24:01.628387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.621328 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgg26\" (UniqueName: \"kubernetes.io/projected/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-kube-api-access-xgg26\") pod \"e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:24:01.628387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.624492 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istio-podinfo\") pod \"e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:24:01.746932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:01.741647 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:24:02.832931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:02.832579 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-461609-g52q5"] Oct 30 00:24:02.848984 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:02.841805 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-461609-g52q5"] Oct 30 00:24:03.103699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:03.103604 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf5f5d9e-2d6f-4861-9279-ce514f89dd86" path="/var/lib/kubelet/pods/cf5f5d9e-2d6f-4861-9279-ce514f89dd86/volumes" Oct 30 00:24:03.169698 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:03.169664 5001 generic.go:356] "Generic (PLEG): container finished" podID="97f84a41-7ac2-4734-ab36-ebf29a62b8fa" containerID="d3be5d676efea82110af81195d4e7082d4262d35d85361fa99390aabeb519fbc" exitCode=0 Oct 30 00:24:03.170275 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:03.170155 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" event={"ID":"97f84a41-7ac2-4734-ab36-ebf29a62b8fa","Type":"ContainerDied","Data":"d3be5d676efea82110af81195d4e7082d4262d35d85361fa99390aabeb519fbc"} Oct 30 00:24:03.170461 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:03.170448 5001 scope.go:122] "RemoveContainer" containerID="4068448d5f41d3e76598637ce899a36296453078dbfb02141c2fbcd30862b98d" Oct 30 00:24:03.940329 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:03.940249 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:24:04.009091 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.009056 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-credential-socket\") pod \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " Oct 30 00:24:04.009286 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.009150 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-workload-socket\") pod \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " Oct 30 00:24:04.009286 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.009213 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istio-token\" (UniqueName: \"kubernetes.io/projected/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istio-token\") pod \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " Oct 30 00:24:04.009286 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.009261 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istio-envoy\") pod \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " Oct 30 00:24:04.009450 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.009425 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-credential-socket" pod "97f84a41-7ac2-4734-ab36-ebf29a62b8fa" (UID: "97f84a41-7ac2-4734-ab36-ebf29a62b8fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:24:04.009514 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.009452 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-workload-certs\") pod \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " Oct 30 00:24:04.009514 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.009506 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istiod-ca-cert\") pod \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " Oct 30 00:24:04.009630 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.009549 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-workload-socket" pod "97f84a41-7ac2-4734-ab36-ebf29a62b8fa" (UID: "97f84a41-7ac2-4734-ab36-ebf29a62b8fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:24:04.009630 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.009556 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-kube-api-access-4dtr2\" (UniqueName: \"kubernetes.io/projected/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-kube-api-access-4dtr2\") pod \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " Oct 30 00:24:04.009747 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.009649 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istio-data\") pod \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " Oct 30 00:24:04.009747 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.009686 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istio-podinfo\") pod \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\" (UID: \"97f84a41-7ac2-4734-ab36-ebf29a62b8fa\") " Oct 30 00:24:04.010972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.010333 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istiod-ca-cert" pod "97f84a41-7ac2-4734-ab36-ebf29a62b8fa" (UID: "97f84a41-7ac2-4734-ab36-ebf29a62b8fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:24:04.010972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.010680 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:24:04.010972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.010702 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:24:04.010972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.010721 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:24:04.010972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.010868 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istio-data" pod "97f84a41-7ac2-4734-ab36-ebf29a62b8fa" (UID: "97f84a41-7ac2-4734-ab36-ebf29a62b8fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:24:04.011905 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.011881 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-workload-certs" pod "97f84a41-7ac2-4734-ab36-ebf29a62b8fa" (UID: "97f84a41-7ac2-4734-ab36-ebf29a62b8fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:24:04.020430 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.020078 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istio-podinfo" pod "97f84a41-7ac2-4734-ab36-ebf29a62b8fa" (UID: "97f84a41-7ac2-4734-ab36-ebf29a62b8fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:24:04.020430 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.020207 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-kube-api-access-4dtr2" pod "97f84a41-7ac2-4734-ab36-ebf29a62b8fa" (UID: "97f84a41-7ac2-4734-ab36-ebf29a62b8fa"). InnerVolumeSpecName "kube-api-access-4dtr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:24:04.020430 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.020277 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istio-token" pod "97f84a41-7ac2-4734-ab36-ebf29a62b8fa" (UID: "97f84a41-7ac2-4734-ab36-ebf29a62b8fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:24:04.021221 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.021177 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istio-envoy" pod "97f84a41-7ac2-4734-ab36-ebf29a62b8fa" (UID: "97f84a41-7ac2-4734-ab36-ebf29a62b8fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:24:04.042261 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.042235 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f4006ca-gwj2w"] Oct 30 00:24:04.050963 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.050898 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f4006ca-gwj2w"] Oct 30 00:24:04.112778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.112722 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:24:04.112778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.112763 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dtr2\" (UniqueName: \"kubernetes.io/projected/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-kube-api-access-4dtr2\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:24:04.112778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.112779 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:24:04.113050 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.112790 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:24:04.113050 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.112802 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:24:04.113050 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.112812 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f84a41-7ac2-4734-ab36-ebf29a62b8fa-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:24:04.184040 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.184004 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" event={"ID":"97f84a41-7ac2-4734-ab36-ebf29a62b8fa","Type":"ContainerDied","Data":"70731f4e20f7530c4f879baad5bff1535877df001a81d3cbd0efe751e24350cb"} Oct 30 00:24:04.184040 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.184028 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8" Oct 30 00:24:04.184281 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.184095 5001 scope.go:122] "RemoveContainer" containerID="d3be5d676efea82110af81195d4e7082d4262d35d85361fa99390aabeb519fbc" Oct 30 00:24:04.236251 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.236219 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8"] Oct 30 00:24:04.242290 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.242265 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-7417dc-openshift-default-5bb7bc885f-2mwj8"] Oct 30 00:24:04.923481 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.923447 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44d3df50-bf97-4d1f-b622-1fbbf7b3734d" path="/var/lib/kubelet/pods/44d3df50-bf97-4d1f-b622-1fbbf7b3734d/volumes" Oct 30 00:24:04.924251 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:04.924231 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97f84a41-7ac2-4734-ab36-ebf29a62b8fa" path="/var/lib/kubelet/pods/97f84a41-7ac2-4734-ab36-ebf29a62b8fa/volumes" Oct 30 00:24:08.488312 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:08.488276 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk"] Oct 30 00:24:08.489091 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:08.489075 5001 cpu_manager.go:414] "containerMap: removing container" podUID="97f84a41-7ac2-4734-ab36-ebf29a62b8fa" containerName="istio-proxy" Oct 30 00:24:08.489152 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:08.489092 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97f84a41-7ac2-4734-ab36-ebf29a62b8fa" containerName="istio-proxy" Oct 30 00:24:08.489152 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:08.489128 5001 cpu_manager.go:414] "containerMap: removing container" podUID="97f84a41-7ac2-4734-ab36-ebf29a62b8fa" containerName="istio-proxy" Oct 30 00:24:08.489152 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:08.489133 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97f84a41-7ac2-4734-ab36-ebf29a62b8fa" containerName="istio-proxy" Oct 30 00:24:08.489245 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:08.489240 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="97f84a41-7ac2-4734-ab36-ebf29a62b8fa" containerName="istio-proxy" Oct 30 00:24:08.489279 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:08.489254 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="97f84a41-7ac2-4734-ab36-ebf29a62b8fa" containerName="istio-proxy" Oct 30 00:24:08.494881 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:08.494858 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" Oct 30 00:24:08.517107 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:08.517079 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk"] Oct 30 00:24:08.521235 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:08.521198 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:24:08.521235 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:08.521203 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:24:08.563002 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:08.562967 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:24:08.563196 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:08.563052 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:24:08.587229 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:08.587193 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1da01660-8624-4707-bd34-edba0f103f2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk\" (UID: \"1da01660-8624-4707-bd34-edba0f103f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" Oct 30 00:24:08.587466 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:08.587449 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p64q\" (UniqueName: \"kubernetes.io/projected/1da01660-8624-4707-bd34-edba0f103f2f-kube-api-access-6p64q\") pod \"maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk\" (UID: \"1da01660-8624-4707-bd34-edba0f103f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" Oct 30 00:24:08.587610 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:08.587593 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1da01660-8624-4707-bd34-edba0f103f2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk\" (UID: \"1da01660-8624-4707-bd34-edba0f103f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" Oct 30 00:24:08.688985 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:08.688949 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1da01660-8624-4707-bd34-edba0f103f2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk\" (UID: \"1da01660-8624-4707-bd34-edba0f103f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" Oct 30 00:24:08.689184 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:08.689056 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1da01660-8624-4707-bd34-edba0f103f2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk\" (UID: \"1da01660-8624-4707-bd34-edba0f103f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" Oct 30 00:24:08.689256 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:08.689223 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p64q\" (UniqueName: \"kubernetes.io/projected/1da01660-8624-4707-bd34-edba0f103f2f-kube-api-access-6p64q\") pod \"maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk\" (UID: \"1da01660-8624-4707-bd34-edba0f103f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" Oct 30 00:24:08.689699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:08.689675 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1da01660-8624-4707-bd34-edba0f103f2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk\" (UID: \"1da01660-8624-4707-bd34-edba0f103f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" Oct 30 00:24:08.691439 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:08.691418 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1da01660-8624-4707-bd34-edba0f103f2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk\" (UID: \"1da01660-8624-4707-bd34-edba0f103f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" Oct 30 00:24:08.710101 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:08.710074 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p64q\" (UniqueName: \"kubernetes.io/projected/1da01660-8624-4707-bd34-edba0f103f2f-kube-api-access-6p64q\") pod \"maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk\" (UID: \"1da01660-8624-4707-bd34-edba0f103f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" Oct 30 00:24:08.811487 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:08.811450 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" Oct 30 00:24:11.421622 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:11.421583 5001 log.go:32] "RunPodSandbox from runtime service failed" err=< Oct 30 00:24:11.421622 ip-10-0-143-120 kubenswrapper[5001]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk_odh-ai-gateway-infra_1da01660-8624-4707-bd34-edba0f103f2f_0(6a2d7fa54fa4143bdd17922a0979618d3fb78974938c7123bbc7dd95b29603d0): error adding pod odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"6a2d7fa54fa4143bdd17922a0979618d3fb78974938c7123bbc7dd95b29603d0" Netns:"/var/run/netns/2b4e460d-ddcd-43a9-ab19-afa4fbc5b287" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk;K8S_POD_INFRA_CONTAINER_ID=6a2d7fa54fa4143bdd17922a0979618d3fb78974938c7123bbc7dd95b29603d0;K8S_POD_UID=1da01660-8624-4707-bd34-edba0f103f2f" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk] networking: Multus: [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk/1da01660-8624-4707-bd34-edba0f103f2f]: error waiting for pod: Get "https://a883217843c784754a9c94a5ff95cefd-0e67250544583792.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk?timeout=1m0s": context deadline exceeded Oct 30 00:24:11.421622 ip-10-0-143-120 kubenswrapper[5001]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 00:24:11.421622 ip-10-0-143-120 kubenswrapper[5001]: > Oct 30 00:24:11.422372 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:11.421650 5001 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Oct 30 00:24:11.422372 ip-10-0-143-120 kubenswrapper[5001]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk_odh-ai-gateway-infra_1da01660-8624-4707-bd34-edba0f103f2f_0(6a2d7fa54fa4143bdd17922a0979618d3fb78974938c7123bbc7dd95b29603d0): error adding pod odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"6a2d7fa54fa4143bdd17922a0979618d3fb78974938c7123bbc7dd95b29603d0" Netns:"/var/run/netns/2b4e460d-ddcd-43a9-ab19-afa4fbc5b287" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk;K8S_POD_INFRA_CONTAINER_ID=6a2d7fa54fa4143bdd17922a0979618d3fb78974938c7123bbc7dd95b29603d0;K8S_POD_UID=1da01660-8624-4707-bd34-edba0f103f2f" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk] networking: Multus: [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk/1da01660-8624-4707-bd34-edba0f103f2f]: error waiting for pod: Get "https://a883217843c784754a9c94a5ff95cefd-0e67250544583792.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk?timeout=1m0s": context deadline exceeded Oct 30 00:24:11.422372 ip-10-0-143-120 kubenswrapper[5001]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 00:24:11.422372 ip-10-0-143-120 kubenswrapper[5001]: > pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" Oct 30 00:24:11.422372 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:11.421674 5001 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=< Oct 30 00:24:11.422372 ip-10-0-143-120 kubenswrapper[5001]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk_odh-ai-gateway-infra_1da01660-8624-4707-bd34-edba0f103f2f_0(6a2d7fa54fa4143bdd17922a0979618d3fb78974938c7123bbc7dd95b29603d0): error adding pod odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"6a2d7fa54fa4143bdd17922a0979618d3fb78974938c7123bbc7dd95b29603d0" Netns:"/var/run/netns/2b4e460d-ddcd-43a9-ab19-afa4fbc5b287" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk;K8S_POD_INFRA_CONTAINER_ID=6a2d7fa54fa4143bdd17922a0979618d3fb78974938c7123bbc7dd95b29603d0;K8S_POD_UID=1da01660-8624-4707-bd34-edba0f103f2f" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk] networking: Multus: [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk/1da01660-8624-4707-bd34-edba0f103f2f]: error waiting for pod: Get "https://a883217843c784754a9c94a5ff95cefd-0e67250544583792.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk?timeout=1m0s": context deadline exceeded Oct 30 00:24:11.422372 ip-10-0-143-120 kubenswrapper[5001]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 00:24:11.422372 ip-10-0-143-120 kubenswrapper[5001]: > pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" Oct 30 00:24:11.422372 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:11.421759 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk_odh-ai-gateway-infra(1da01660-8624-4707-bd34-edba0f103f2f)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk_odh-ai-gateway-infra(1da01660-8624-4707-bd34-edba0f103f2f)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk_odh-ai-gateway-infra_1da01660-8624-4707-bd34-edba0f103f2f_0(6a2d7fa54fa4143bdd17922a0979618d3fb78974938c7123bbc7dd95b29603d0): error adding pod odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"6a2d7fa54fa4143bdd17922a0979618d3fb78974938c7123bbc7dd95b29603d0\\\" Netns:\\\"/var/run/netns/2b4e460d-ddcd-43a9-ab19-afa4fbc5b287\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk;K8S_POD_INFRA_CONTAINER_ID=6a2d7fa54fa4143bdd17922a0979618d3fb78974938c7123bbc7dd95b29603d0;K8S_POD_UID=1da01660-8624-4707-bd34-edba0f103f2f\\\" Path:\\\"\\\" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk] networking: Multus: [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk/1da01660-8624-4707-bd34-edba0f103f2f]: error waiting for pod: Get \\\"https://a883217843c784754a9c94a5ff95cefd-0e67250544583792.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" podUID="1da01660-8624-4707-bd34-edba0f103f2f" Oct 30 00:24:12.348533 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:12.348497 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" Oct 30 00:24:12.348898 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:12.348885 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" Oct 30 00:24:13.029743 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:13.029706 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3675dc8c-h57sw"] Oct 30 00:24:13.038973 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:13.038938 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3675dc8c-h57sw"] Oct 30 00:24:14.920724 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:14.920689 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2123cf26-a896-471e-9d46-7916c242407c" path="/var/lib/kubelet/pods/2123cf26-a896-471e-9d46-7916c242407c/volumes" Oct 30 00:24:14.954904 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:14.954873 5001 log.go:32] "RunPodSandbox from runtime service failed" err=< Oct 30 00:24:14.954904 ip-10-0-143-120 kubenswrapper[5001]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk_odh-ai-gateway-infra_1da01660-8624-4707-bd34-edba0f103f2f_0(98b33603b33a9bafcd16b1b5f2532ba4ac316f29958b14b3d233a8b6d7c7773a): error adding pod odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"98b33603b33a9bafcd16b1b5f2532ba4ac316f29958b14b3d233a8b6d7c7773a" Netns:"/var/run/netns/6fa207d0-f1cb-4f2b-a0a1-086a1cb3336e" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk;K8S_POD_INFRA_CONTAINER_ID=98b33603b33a9bafcd16b1b5f2532ba4ac316f29958b14b3d233a8b6d7c7773a;K8S_POD_UID=1da01660-8624-4707-bd34-edba0f103f2f" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk] networking: Multus: [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk/1da01660-8624-4707-bd34-edba0f103f2f]: error waiting for pod: Get "https://a883217843c784754a9c94a5ff95cefd-0e67250544583792.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk?timeout=1m0s": context deadline exceeded Oct 30 00:24:14.954904 ip-10-0-143-120 kubenswrapper[5001]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 00:24:14.954904 ip-10-0-143-120 kubenswrapper[5001]: > Oct 30 00:24:14.955473 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:14.954979 5001 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Oct 30 00:24:14.955473 ip-10-0-143-120 kubenswrapper[5001]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk_odh-ai-gateway-infra_1da01660-8624-4707-bd34-edba0f103f2f_0(98b33603b33a9bafcd16b1b5f2532ba4ac316f29958b14b3d233a8b6d7c7773a): error adding pod odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"98b33603b33a9bafcd16b1b5f2532ba4ac316f29958b14b3d233a8b6d7c7773a" Netns:"/var/run/netns/6fa207d0-f1cb-4f2b-a0a1-086a1cb3336e" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk;K8S_POD_INFRA_CONTAINER_ID=98b33603b33a9bafcd16b1b5f2532ba4ac316f29958b14b3d233a8b6d7c7773a;K8S_POD_UID=1da01660-8624-4707-bd34-edba0f103f2f" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk] networking: Multus: [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk/1da01660-8624-4707-bd34-edba0f103f2f]: error waiting for pod: Get "https://a883217843c784754a9c94a5ff95cefd-0e67250544583792.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk?timeout=1m0s": context deadline exceeded Oct 30 00:24:14.955473 ip-10-0-143-120 kubenswrapper[5001]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 00:24:14.955473 ip-10-0-143-120 kubenswrapper[5001]: > pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" Oct 30 00:24:14.955473 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:14.955001 5001 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=< Oct 30 00:24:14.955473 ip-10-0-143-120 kubenswrapper[5001]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk_odh-ai-gateway-infra_1da01660-8624-4707-bd34-edba0f103f2f_0(98b33603b33a9bafcd16b1b5f2532ba4ac316f29958b14b3d233a8b6d7c7773a): error adding pod odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"98b33603b33a9bafcd16b1b5f2532ba4ac316f29958b14b3d233a8b6d7c7773a" Netns:"/var/run/netns/6fa207d0-f1cb-4f2b-a0a1-086a1cb3336e" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk;K8S_POD_INFRA_CONTAINER_ID=98b33603b33a9bafcd16b1b5f2532ba4ac316f29958b14b3d233a8b6d7c7773a;K8S_POD_UID=1da01660-8624-4707-bd34-edba0f103f2f" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk] networking: Multus: [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk/1da01660-8624-4707-bd34-edba0f103f2f]: error waiting for pod: Get "https://a883217843c784754a9c94a5ff95cefd-0e67250544583792.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk?timeout=1m0s": context deadline exceeded Oct 30 00:24:14.955473 ip-10-0-143-120 kubenswrapper[5001]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 00:24:14.955473 ip-10-0-143-120 kubenswrapper[5001]: > pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" Oct 30 00:24:14.955473 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:14.955077 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk_odh-ai-gateway-infra(1da01660-8624-4707-bd34-edba0f103f2f)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk_odh-ai-gateway-infra(1da01660-8624-4707-bd34-edba0f103f2f)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk_odh-ai-gateway-infra_1da01660-8624-4707-bd34-edba0f103f2f_0(98b33603b33a9bafcd16b1b5f2532ba4ac316f29958b14b3d233a8b6d7c7773a): error adding pod odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"98b33603b33a9bafcd16b1b5f2532ba4ac316f29958b14b3d233a8b6d7c7773a\\\" Netns:\\\"/var/run/netns/6fa207d0-f1cb-4f2b-a0a1-086a1cb3336e\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk;K8S_POD_INFRA_CONTAINER_ID=98b33603b33a9bafcd16b1b5f2532ba4ac316f29958b14b3d233a8b6d7c7773a;K8S_POD_UID=1da01660-8624-4707-bd34-edba0f103f2f\\\" Path:\\\"\\\" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk] networking: Multus: [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk/1da01660-8624-4707-bd34-edba0f103f2f]: error waiting for pod: Get \\\"https://a883217843c784754a9c94a5ff95cefd-0e67250544583792.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" podUID="1da01660-8624-4707-bd34-edba0f103f2f" Oct 30 00:24:17.324697 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:17.324655 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:24:20.107268 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:20.107234 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp"] Oct 30 00:24:20.107646 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:20.107486 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" podUID="1584ba56-ffef-49e1-8edd-399ca7b7f366" containerName="istio-proxy" containerID="cri-o://4ab18249e82ad6665281eaa53ea83fd1ba4eb61eee71e445a0700225c4d9d504" gracePeriod=30 Oct 30 00:24:20.726025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:20.725978 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78"] Oct 30 00:24:20.761273 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:20.761242 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78" Oct 30 00:24:20.763371 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:20.763338 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78"] Oct 30 00:24:20.831934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:20.799466 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccde7ce6-b81a-4564-a8f2-15669279f5a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-08dd33-zzt78\" (UID: \"ccde7ce6-b81a-4564-a8f2-15669279f5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78" Oct 30 00:24:20.832546 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:20.832498 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccde7ce6-b81a-4564-a8f2-15669279f5a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-08dd33-zzt78\" (UID: \"ccde7ce6-b81a-4564-a8f2-15669279f5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78" Oct 30 00:24:20.833087 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:20.833053 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6bs7\" (UniqueName: \"kubernetes.io/projected/ccde7ce6-b81a-4564-a8f2-15669279f5a9-kube-api-access-b6bs7\") pod \"maas-api-revoke-keys-e2e-api-b-08dd33-zzt78\" (UID: \"ccde7ce6-b81a-4564-a8f2-15669279f5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78" Oct 30 00:24:21.012713 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:20.982158 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6bs7\" (UniqueName: \"kubernetes.io/projected/ccde7ce6-b81a-4564-a8f2-15669279f5a9-kube-api-access-b6bs7\") pod \"maas-api-revoke-keys-e2e-api-b-08dd33-zzt78\" (UID: \"ccde7ce6-b81a-4564-a8f2-15669279f5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78" Oct 30 00:24:21.012713 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:20.982259 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccde7ce6-b81a-4564-a8f2-15669279f5a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-08dd33-zzt78\" (UID: \"ccde7ce6-b81a-4564-a8f2-15669279f5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78" Oct 30 00:24:21.012713 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:20.982485 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccde7ce6-b81a-4564-a8f2-15669279f5a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-08dd33-zzt78\" (UID: \"ccde7ce6-b81a-4564-a8f2-15669279f5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78" Oct 30 00:24:21.012713 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:20.989476 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccde7ce6-b81a-4564-a8f2-15669279f5a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-08dd33-zzt78\" (UID: \"ccde7ce6-b81a-4564-a8f2-15669279f5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78" Oct 30 00:24:21.017832 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:21.016740 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6bs7\" (UniqueName: \"kubernetes.io/projected/ccde7ce6-b81a-4564-a8f2-15669279f5a9-kube-api-access-b6bs7\") pod \"maas-api-revoke-keys-e2e-api-b-08dd33-zzt78\" (UID: \"ccde7ce6-b81a-4564-a8f2-15669279f5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78" Oct 30 00:24:21.377381 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:21.377065 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccde7ce6-b81a-4564-a8f2-15669279f5a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-08dd33-zzt78\" (UID: \"ccde7ce6-b81a-4564-a8f2-15669279f5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78" Oct 30 00:24:21.712984 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:21.708559 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78" Oct 30 00:24:22.025995 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:22.024444 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:24:23.415119 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:23.415057 5001 patch_prober.go:29] interesting pod/e2e-api-a-461609-openshift-default-86bb859445-gnfjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:24:23.415690 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:23.415659 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" podUID="1584ba56-ffef-49e1-8edd-399ca7b7f366" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:24:25.033016 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:25.031443 5001 log.go:32] "RunPodSandbox from runtime service failed" err=< Oct 30 00:24:25.033016 ip-10-0-143-120 kubenswrapper[5001]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-revoke-keys-e2e-api-b-08dd33-zzt78_odh-ai-gateway-infra_ccde7ce6-b81a-4564-a8f2-15669279f5a9_0(83a145e1d674b321f952172599c5ade6d5dc8aee6fcd866552c968a376f1287a): error adding pod odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-08dd33-zzt78 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"83a145e1d674b321f952172599c5ade6d5dc8aee6fcd866552c968a376f1287a" Netns:"/var/run/netns/08f8d21b-7624-470d-8847-a77108481715" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-revoke-keys-e2e-api-b-08dd33-zzt78;K8S_POD_INFRA_CONTAINER_ID=83a145e1d674b321f952172599c5ade6d5dc8aee6fcd866552c968a376f1287a;K8S_POD_UID=ccde7ce6-b81a-4564-a8f2-15669279f5a9" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78] networking: Multus: [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78/ccde7ce6-b81a-4564-a8f2-15669279f5a9]: error waiting for pod: Get "https://a883217843c784754a9c94a5ff95cefd-0e67250544583792.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78?timeout=1m0s": context deadline exceeded Oct 30 00:24:25.033016 ip-10-0-143-120 kubenswrapper[5001]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 00:24:25.033016 ip-10-0-143-120 kubenswrapper[5001]: > Oct 30 00:24:25.033016 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:25.031500 5001 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Oct 30 00:24:25.033016 ip-10-0-143-120 kubenswrapper[5001]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-revoke-keys-e2e-api-b-08dd33-zzt78_odh-ai-gateway-infra_ccde7ce6-b81a-4564-a8f2-15669279f5a9_0(83a145e1d674b321f952172599c5ade6d5dc8aee6fcd866552c968a376f1287a): error adding pod odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-08dd33-zzt78 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"83a145e1d674b321f952172599c5ade6d5dc8aee6fcd866552c968a376f1287a" Netns:"/var/run/netns/08f8d21b-7624-470d-8847-a77108481715" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-revoke-keys-e2e-api-b-08dd33-zzt78;K8S_POD_INFRA_CONTAINER_ID=83a145e1d674b321f952172599c5ade6d5dc8aee6fcd866552c968a376f1287a;K8S_POD_UID=ccde7ce6-b81a-4564-a8f2-15669279f5a9" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78] networking: Multus: [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78/ccde7ce6-b81a-4564-a8f2-15669279f5a9]: error waiting for pod: Get "https://a883217843c784754a9c94a5ff95cefd-0e67250544583792.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78?timeout=1m0s": context deadline exceeded Oct 30 00:24:25.033016 ip-10-0-143-120 kubenswrapper[5001]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 00:24:25.033016 ip-10-0-143-120 kubenswrapper[5001]: > pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78" Oct 30 00:24:25.033016 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:25.031523 5001 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=< Oct 30 00:24:25.033016 ip-10-0-143-120 kubenswrapper[5001]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-revoke-keys-e2e-api-b-08dd33-zzt78_odh-ai-gateway-infra_ccde7ce6-b81a-4564-a8f2-15669279f5a9_0(83a145e1d674b321f952172599c5ade6d5dc8aee6fcd866552c968a376f1287a): error adding pod odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-08dd33-zzt78 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"83a145e1d674b321f952172599c5ade6d5dc8aee6fcd866552c968a376f1287a" Netns:"/var/run/netns/08f8d21b-7624-470d-8847-a77108481715" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-revoke-keys-e2e-api-b-08dd33-zzt78;K8S_POD_INFRA_CONTAINER_ID=83a145e1d674b321f952172599c5ade6d5dc8aee6fcd866552c968a376f1287a;K8S_POD_UID=ccde7ce6-b81a-4564-a8f2-15669279f5a9" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78] networking: Multus: [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78/ccde7ce6-b81a-4564-a8f2-15669279f5a9]: error waiting for pod: Get "https://a883217843c784754a9c94a5ff95cefd-0e67250544583792.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78?timeout=1m0s": context deadline exceeded Oct 30 00:24:25.033016 ip-10-0-143-120 kubenswrapper[5001]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 00:24:25.033016 ip-10-0-143-120 kubenswrapper[5001]: > pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78" Oct 30 00:24:25.033016 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:25.031618 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-api-revoke-keys-e2e-api-b-08dd33-zzt78_odh-ai-gateway-infra(ccde7ce6-b81a-4564-a8f2-15669279f5a9)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-api-revoke-keys-e2e-api-b-08dd33-zzt78_odh-ai-gateway-infra(ccde7ce6-b81a-4564-a8f2-15669279f5a9)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-revoke-keys-e2e-api-b-08dd33-zzt78_odh-ai-gateway-infra_ccde7ce6-b81a-4564-a8f2-15669279f5a9_0(83a145e1d674b321f952172599c5ade6d5dc8aee6fcd866552c968a376f1287a): error adding pod odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-08dd33-zzt78 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"83a145e1d674b321f952172599c5ade6d5dc8aee6fcd866552c968a376f1287a\\\" Netns:\\\"/var/run/netns/08f8d21b-7624-470d-8847-a77108481715\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-revoke-keys-e2e-api-b-08dd33-zzt78;K8S_POD_INFRA_CONTAINER_ID=83a145e1d674b321f952172599c5ade6d5dc8aee6fcd866552c968a376f1287a;K8S_POD_UID=ccde7ce6-b81a-4564-a8f2-15669279f5a9\\\" Path:\\\"\\\" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78] networking: Multus: [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78/ccde7ce6-b81a-4564-a8f2-15669279f5a9]: error waiting for pod: Get \\\"https://a883217843c784754a9c94a5ff95cefd-0e67250544583792.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78" podUID="ccde7ce6-b81a-4564-a8f2-15669279f5a9" Oct 30 00:24:25.450962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:25.450891 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:24:25.543822 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:25.543746 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-workload-socket\") pod \"1584ba56-ffef-49e1-8edd-399ca7b7f366\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " Oct 30 00:24:25.543822 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:25.543807 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1584ba56-ffef-49e1-8edd-399ca7b7f366-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1584ba56-ffef-49e1-8edd-399ca7b7f366-istio-podinfo\") pod \"1584ba56-ffef-49e1-8edd-399ca7b7f366\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " Oct 30 00:24:25.544100 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:25.543842 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-workload-certs\") pod \"1584ba56-ffef-49e1-8edd-399ca7b7f366\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " Oct 30 00:24:25.544100 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:25.543878 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-credential-socket\") pod \"1584ba56-ffef-49e1-8edd-399ca7b7f366\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " Oct 30 00:24:25.544100 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:25.544026 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-istio-envoy\") pod \"1584ba56-ffef-49e1-8edd-399ca7b7f366\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " Oct 30 00:24:25.544100 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:25.544044 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-workload-socket" pod "1584ba56-ffef-49e1-8edd-399ca7b7f366" (UID: "1584ba56-ffef-49e1-8edd-399ca7b7f366"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:24:25.544327 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:25.544132 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-istio-data\") pod \"1584ba56-ffef-49e1-8edd-399ca7b7f366\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " Oct 30 00:24:25.544327 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:25.544212 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1584ba56-ffef-49e1-8edd-399ca7b7f366-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1584ba56-ffef-49e1-8edd-399ca7b7f366-istiod-ca-cert\") pod \"1584ba56-ffef-49e1-8edd-399ca7b7f366\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " Oct 30 00:24:25.544327 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:25.544310 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1584ba56-ffef-49e1-8edd-399ca7b7f366-istio-token\" (UniqueName: \"kubernetes.io/projected/1584ba56-ffef-49e1-8edd-399ca7b7f366-istio-token\") pod \"1584ba56-ffef-49e1-8edd-399ca7b7f366\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " Oct 30 00:24:25.544852 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:25.544512 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1584ba56-ffef-49e1-8edd-399ca7b7f366-kube-api-access-7cxxq\" (UniqueName: \"kubernetes.io/projected/1584ba56-ffef-49e1-8edd-399ca7b7f366-kube-api-access-7cxxq\") pod \"1584ba56-ffef-49e1-8edd-399ca7b7f366\" (UID: \"1584ba56-ffef-49e1-8edd-399ca7b7f366\") " Oct 30 00:24:25.544852 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:25.544216 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-workload-certs" pod "1584ba56-ffef-49e1-8edd-399ca7b7f366" (UID: "1584ba56-ffef-49e1-8edd-399ca7b7f366"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:24:25.544852 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:25.544685 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-istio-data" pod "1584ba56-ffef-49e1-8edd-399ca7b7f366" (UID: "1584ba56-ffef-49e1-8edd-399ca7b7f366"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:24:25.544852 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:25.544742 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-credential-socket" pod "1584ba56-ffef-49e1-8edd-399ca7b7f366" (UID: "1584ba56-ffef-49e1-8edd-399ca7b7f366"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:24:25.546253 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:25.545479 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1584ba56-ffef-49e1-8edd-399ca7b7f366-istiod-ca-cert" pod "1584ba56-ffef-49e1-8edd-399ca7b7f366" (UID: "1584ba56-ffef-49e1-8edd-399ca7b7f366"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:24:25.546253 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:25.546141 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:24:25.546253 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:25.546163 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:24:25.546253 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:25.546179 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:24:25.546253 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:25.546192 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:24:25.546253 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:25.546205 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1584ba56-ffef-49e1-8edd-399ca7b7f366-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:24:25.546830 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:25.546809 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1584ba56-ffef-49e1-8edd-399ca7b7f366-istio-podinfo" pod "1584ba56-ffef-49e1-8edd-399ca7b7f366" (UID: "1584ba56-ffef-49e1-8edd-399ca7b7f366"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:24:25.547165 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:25.547146 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-istio-envoy" pod "1584ba56-ffef-49e1-8edd-399ca7b7f366" (UID: "1584ba56-ffef-49e1-8edd-399ca7b7f366"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:24:25.547456 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:25.547434 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1584ba56-ffef-49e1-8edd-399ca7b7f366-kube-api-access-7cxxq" pod "1584ba56-ffef-49e1-8edd-399ca7b7f366" (UID: "1584ba56-ffef-49e1-8edd-399ca7b7f366"). InnerVolumeSpecName "kube-api-access-7cxxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:24:25.547691 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:25.547673 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1584ba56-ffef-49e1-8edd-399ca7b7f366-istio-token" pod "1584ba56-ffef-49e1-8edd-399ca7b7f366" (UID: "1584ba56-ffef-49e1-8edd-399ca7b7f366"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:24:25.647461 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:25.647419 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1584ba56-ffef-49e1-8edd-399ca7b7f366-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:24:25.647461 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:25.647456 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cxxq\" (UniqueName: \"kubernetes.io/projected/1584ba56-ffef-49e1-8edd-399ca7b7f366-kube-api-access-7cxxq\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:24:25.647461 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:25.647472 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1584ba56-ffef-49e1-8edd-399ca7b7f366-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:24:25.647731 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:25.647485 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1584ba56-ffef-49e1-8edd-399ca7b7f366-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:24:25.916932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:25.915166 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" Oct 30 00:24:25.916932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:25.915660 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" Oct 30 00:24:26.053994 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:26.053957 5001 generic.go:356] "Generic (PLEG): container finished" podID="1584ba56-ffef-49e1-8edd-399ca7b7f366" containerID="4ab18249e82ad6665281eaa53ea83fd1ba4eb61eee71e445a0700225c4d9d504" exitCode=0 Oct 30 00:24:26.054800 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:26.054764 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" event={"ID":"1584ba56-ffef-49e1-8edd-399ca7b7f366","Type":"ContainerDied","Data":"4ab18249e82ad6665281eaa53ea83fd1ba4eb61eee71e445a0700225c4d9d504"} Oct 30 00:24:26.054979 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:26.054927 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78" Oct 30 00:24:26.055051 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:26.054785 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" Oct 30 00:24:26.055051 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:26.055019 5001 scope.go:122] "RemoveContainer" containerID="4ab18249e82ad6665281eaa53ea83fd1ba4eb61eee71e445a0700225c4d9d504" Oct 30 00:24:26.055195 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:26.054998 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp" event={"ID":"1584ba56-ffef-49e1-8edd-399ca7b7f366","Type":"ContainerDied","Data":"c50c4d02603cc2266adeb954918be7ac8c47c393aff415d48b4deb21339f414a"} Oct 30 00:24:26.055736 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:26.055721 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78" Oct 30 00:24:26.079537 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:26.079515 5001 scope.go:122] "RemoveContainer" containerID="68562c9c74560a1ba68b6a3474a65cfdebcc91dd3793ba80d25924a386d06e26" Oct 30 00:24:26.139029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:26.138997 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp"] Oct 30 00:24:26.141052 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:26.141030 5001 scope.go:122] "RemoveContainer" containerID="4ab18249e82ad6665281eaa53ea83fd1ba4eb61eee71e445a0700225c4d9d504" Oct 30 00:24:26.141399 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:26.141374 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ab18249e82ad6665281eaa53ea83fd1ba4eb61eee71e445a0700225c4d9d504\": container with ID starting with 4ab18249e82ad6665281eaa53ea83fd1ba4eb61eee71e445a0700225c4d9d504 not found: ID does not exist" containerID="4ab18249e82ad6665281eaa53ea83fd1ba4eb61eee71e445a0700225c4d9d504" Oct 30 00:24:26.141534 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:26.141406 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ab18249e82ad6665281eaa53ea83fd1ba4eb61eee71e445a0700225c4d9d504"} err="failed to get container status \"4ab18249e82ad6665281eaa53ea83fd1ba4eb61eee71e445a0700225c4d9d504\": rpc error: code = NotFound desc = could not find container \"4ab18249e82ad6665281eaa53ea83fd1ba4eb61eee71e445a0700225c4d9d504\": container with ID starting with 4ab18249e82ad6665281eaa53ea83fd1ba4eb61eee71e445a0700225c4d9d504 not found: ID does not exist" Oct 30 00:24:26.141534 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:26.141426 5001 scope.go:122] "RemoveContainer" containerID="68562c9c74560a1ba68b6a3474a65cfdebcc91dd3793ba80d25924a386d06e26" Oct 30 00:24:26.141744 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:26.141697 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68562c9c74560a1ba68b6a3474a65cfdebcc91dd3793ba80d25924a386d06e26\": container with ID starting with 68562c9c74560a1ba68b6a3474a65cfdebcc91dd3793ba80d25924a386d06e26 not found: ID does not exist" containerID="68562c9c74560a1ba68b6a3474a65cfdebcc91dd3793ba80d25924a386d06e26" Oct 30 00:24:26.141744 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:26.141722 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68562c9c74560a1ba68b6a3474a65cfdebcc91dd3793ba80d25924a386d06e26"} err="failed to get container status \"68562c9c74560a1ba68b6a3474a65cfdebcc91dd3793ba80d25924a386d06e26\": rpc error: code = NotFound desc = could not find container \"68562c9c74560a1ba68b6a3474a65cfdebcc91dd3793ba80d25924a386d06e26\": container with ID starting with 68562c9c74560a1ba68b6a3474a65cfdebcc91dd3793ba80d25924a386d06e26 not found: ID does not exist" Oct 30 00:24:26.144839 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:26.144813 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-461609-openshift-default-86bb859445-gnfjp"] Oct 30 00:24:26.991941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:26.988656 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1584ba56-ffef-49e1-8edd-399ca7b7f366" path="/var/lib/kubelet/pods/1584ba56-ffef-49e1-8edd-399ca7b7f366/volumes" Oct 30 00:24:27.089165 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:27.089072 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56e7f57d-p8g94"] Oct 30 00:24:27.288935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:27.287045 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56e7f57d-p8g94"] Oct 30 00:24:28.546093 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:28.546055 5001 log.go:32] "RunPodSandbox from runtime service failed" err=< Oct 30 00:24:28.546093 ip-10-0-143-120 kubenswrapper[5001]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk_odh-ai-gateway-infra_1da01660-8624-4707-bd34-edba0f103f2f_0(d30663a84795c859d1481b03656bfde3a91ddf55bcd514420ee38e63c6284bd3): error adding pod odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"d30663a84795c859d1481b03656bfde3a91ddf55bcd514420ee38e63c6284bd3" Netns:"/var/run/netns/1e49efbe-99b4-4800-b4ec-ad412a486631" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk;K8S_POD_INFRA_CONTAINER_ID=d30663a84795c859d1481b03656bfde3a91ddf55bcd514420ee38e63c6284bd3;K8S_POD_UID=1da01660-8624-4707-bd34-edba0f103f2f" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk] networking: Multus: [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk/1da01660-8624-4707-bd34-edba0f103f2f]: error waiting for pod: Get "https://a883217843c784754a9c94a5ff95cefd-0e67250544583792.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk?timeout=1m0s": context deadline exceeded Oct 30 00:24:28.546093 ip-10-0-143-120 kubenswrapper[5001]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 00:24:28.546093 ip-10-0-143-120 kubenswrapper[5001]: > Oct 30 00:24:28.546574 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:28.546118 5001 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Oct 30 00:24:28.546574 ip-10-0-143-120 kubenswrapper[5001]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk_odh-ai-gateway-infra_1da01660-8624-4707-bd34-edba0f103f2f_0(d30663a84795c859d1481b03656bfde3a91ddf55bcd514420ee38e63c6284bd3): error adding pod odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"d30663a84795c859d1481b03656bfde3a91ddf55bcd514420ee38e63c6284bd3" Netns:"/var/run/netns/1e49efbe-99b4-4800-b4ec-ad412a486631" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk;K8S_POD_INFRA_CONTAINER_ID=d30663a84795c859d1481b03656bfde3a91ddf55bcd514420ee38e63c6284bd3;K8S_POD_UID=1da01660-8624-4707-bd34-edba0f103f2f" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk] networking: Multus: [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk/1da01660-8624-4707-bd34-edba0f103f2f]: error waiting for pod: Get "https://a883217843c784754a9c94a5ff95cefd-0e67250544583792.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk?timeout=1m0s": context deadline exceeded Oct 30 00:24:28.546574 ip-10-0-143-120 kubenswrapper[5001]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 00:24:28.546574 ip-10-0-143-120 kubenswrapper[5001]: > pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" Oct 30 00:24:28.546574 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:28.546138 5001 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=< Oct 30 00:24:28.546574 ip-10-0-143-120 kubenswrapper[5001]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk_odh-ai-gateway-infra_1da01660-8624-4707-bd34-edba0f103f2f_0(d30663a84795c859d1481b03656bfde3a91ddf55bcd514420ee38e63c6284bd3): error adding pod odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"d30663a84795c859d1481b03656bfde3a91ddf55bcd514420ee38e63c6284bd3" Netns:"/var/run/netns/1e49efbe-99b4-4800-b4ec-ad412a486631" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk;K8S_POD_INFRA_CONTAINER_ID=d30663a84795c859d1481b03656bfde3a91ddf55bcd514420ee38e63c6284bd3;K8S_POD_UID=1da01660-8624-4707-bd34-edba0f103f2f" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk] networking: Multus: [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk/1da01660-8624-4707-bd34-edba0f103f2f]: error waiting for pod: Get "https://a883217843c784754a9c94a5ff95cefd-0e67250544583792.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk?timeout=1m0s": context deadline exceeded Oct 30 00:24:28.546574 ip-10-0-143-120 kubenswrapper[5001]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 00:24:28.546574 ip-10-0-143-120 kubenswrapper[5001]: > pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" Oct 30 00:24:28.546574 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:28.546221 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk_odh-ai-gateway-infra(1da01660-8624-4707-bd34-edba0f103f2f)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk_odh-ai-gateway-infra(1da01660-8624-4707-bd34-edba0f103f2f)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk_odh-ai-gateway-infra_1da01660-8624-4707-bd34-edba0f103f2f_0(d30663a84795c859d1481b03656bfde3a91ddf55bcd514420ee38e63c6284bd3): error adding pod odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"d30663a84795c859d1481b03656bfde3a91ddf55bcd514420ee38e63c6284bd3\\\" Netns:\\\"/var/run/netns/1e49efbe-99b4-4800-b4ec-ad412a486631\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk;K8S_POD_INFRA_CONTAINER_ID=d30663a84795c859d1481b03656bfde3a91ddf55bcd514420ee38e63c6284bd3;K8S_POD_UID=1da01660-8624-4707-bd34-edba0f103f2f\\\" Path:\\\"\\\" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk] networking: Multus: [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk/1da01660-8624-4707-bd34-edba0f103f2f]: error waiting for pod: Get \\\"https://a883217843c784754a9c94a5ff95cefd-0e67250544583792.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" podUID="1da01660-8624-4707-bd34-edba0f103f2f" Oct 30 00:24:28.668881 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:28.668852 5001 log.go:32] "RunPodSandbox from runtime service failed" err=< Oct 30 00:24:28.668881 ip-10-0-143-120 kubenswrapper[5001]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-revoke-keys-e2e-api-b-08dd33-zzt78_odh-ai-gateway-infra_ccde7ce6-b81a-4564-a8f2-15669279f5a9_0(8452e9b66403a0ff15fa0258d5177a11c845f90162e2cb2708d06afa40419885): error adding pod odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-08dd33-zzt78 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"8452e9b66403a0ff15fa0258d5177a11c845f90162e2cb2708d06afa40419885" Netns:"/var/run/netns/60b18001-4ca6-41ff-958f-3f10e53f0da6" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-revoke-keys-e2e-api-b-08dd33-zzt78;K8S_POD_INFRA_CONTAINER_ID=8452e9b66403a0ff15fa0258d5177a11c845f90162e2cb2708d06afa40419885;K8S_POD_UID=ccde7ce6-b81a-4564-a8f2-15669279f5a9" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78] networking: Multus: [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78/ccde7ce6-b81a-4564-a8f2-15669279f5a9]: error waiting for pod: Get "https://a883217843c784754a9c94a5ff95cefd-0e67250544583792.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78?timeout=1m0s": context deadline exceeded Oct 30 00:24:28.668881 ip-10-0-143-120 kubenswrapper[5001]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 00:24:28.668881 ip-10-0-143-120 kubenswrapper[5001]: > Oct 30 00:24:28.669086 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:28.668902 5001 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Oct 30 00:24:28.669086 ip-10-0-143-120 kubenswrapper[5001]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-revoke-keys-e2e-api-b-08dd33-zzt78_odh-ai-gateway-infra_ccde7ce6-b81a-4564-a8f2-15669279f5a9_0(8452e9b66403a0ff15fa0258d5177a11c845f90162e2cb2708d06afa40419885): error adding pod odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-08dd33-zzt78 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"8452e9b66403a0ff15fa0258d5177a11c845f90162e2cb2708d06afa40419885" Netns:"/var/run/netns/60b18001-4ca6-41ff-958f-3f10e53f0da6" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-revoke-keys-e2e-api-b-08dd33-zzt78;K8S_POD_INFRA_CONTAINER_ID=8452e9b66403a0ff15fa0258d5177a11c845f90162e2cb2708d06afa40419885;K8S_POD_UID=ccde7ce6-b81a-4564-a8f2-15669279f5a9" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78] networking: Multus: [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78/ccde7ce6-b81a-4564-a8f2-15669279f5a9]: error waiting for pod: Get "https://a883217843c784754a9c94a5ff95cefd-0e67250544583792.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78?timeout=1m0s": context deadline exceeded Oct 30 00:24:28.669086 ip-10-0-143-120 kubenswrapper[5001]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 00:24:28.669086 ip-10-0-143-120 kubenswrapper[5001]: > pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78" Oct 30 00:24:28.669086 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:28.668950 5001 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=< Oct 30 00:24:28.669086 ip-10-0-143-120 kubenswrapper[5001]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-revoke-keys-e2e-api-b-08dd33-zzt78_odh-ai-gateway-infra_ccde7ce6-b81a-4564-a8f2-15669279f5a9_0(8452e9b66403a0ff15fa0258d5177a11c845f90162e2cb2708d06afa40419885): error adding pod odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-08dd33-zzt78 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"8452e9b66403a0ff15fa0258d5177a11c845f90162e2cb2708d06afa40419885" Netns:"/var/run/netns/60b18001-4ca6-41ff-958f-3f10e53f0da6" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-revoke-keys-e2e-api-b-08dd33-zzt78;K8S_POD_INFRA_CONTAINER_ID=8452e9b66403a0ff15fa0258d5177a11c845f90162e2cb2708d06afa40419885;K8S_POD_UID=ccde7ce6-b81a-4564-a8f2-15669279f5a9" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78] networking: Multus: [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78/ccde7ce6-b81a-4564-a8f2-15669279f5a9]: error waiting for pod: Get "https://a883217843c784754a9c94a5ff95cefd-0e67250544583792.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78?timeout=1m0s": context deadline exceeded Oct 30 00:24:28.669086 ip-10-0-143-120 kubenswrapper[5001]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 00:24:28.669086 ip-10-0-143-120 kubenswrapper[5001]: > pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78" Oct 30 00:24:28.669086 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:28.669038 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-api-revoke-keys-e2e-api-b-08dd33-zzt78_odh-ai-gateway-infra(ccde7ce6-b81a-4564-a8f2-15669279f5a9)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-api-revoke-keys-e2e-api-b-08dd33-zzt78_odh-ai-gateway-infra(ccde7ce6-b81a-4564-a8f2-15669279f5a9)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-revoke-keys-e2e-api-b-08dd33-zzt78_odh-ai-gateway-infra_ccde7ce6-b81a-4564-a8f2-15669279f5a9_0(8452e9b66403a0ff15fa0258d5177a11c845f90162e2cb2708d06afa40419885): error adding pod odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-08dd33-zzt78 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"8452e9b66403a0ff15fa0258d5177a11c845f90162e2cb2708d06afa40419885\\\" Netns:\\\"/var/run/netns/60b18001-4ca6-41ff-958f-3f10e53f0da6\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-revoke-keys-e2e-api-b-08dd33-zzt78;K8S_POD_INFRA_CONTAINER_ID=8452e9b66403a0ff15fa0258d5177a11c845f90162e2cb2708d06afa40419885;K8S_POD_UID=ccde7ce6-b81a-4564-a8f2-15669279f5a9\\\" Path:\\\"\\\" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78] networking: Multus: [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78/ccde7ce6-b81a-4564-a8f2-15669279f5a9]: error waiting for pod: Get \\\"https://a883217843c784754a9c94a5ff95cefd-0e67250544583792.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78" podUID="ccde7ce6-b81a-4564-a8f2-15669279f5a9" Oct 30 00:24:28.933240 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:28.932510 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d42bb9aa-fb91-462a-a01f-01c821cc2bb6" path="/var/lib/kubelet/pods/d42bb9aa-fb91-462a-a01f-01c821cc2bb6/volumes" Oct 30 00:24:29.048158 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:29.048129 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f31177a-9fv92"] Oct 30 00:24:29.058368 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:29.058338 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f31177a-9fv92"] Oct 30 00:24:30.920533 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:30.920499 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e1a4c0b-7bda-4ada-8221-bd444286bd94" path="/var/lib/kubelet/pods/1e1a4c0b-7bda-4ada-8221-bd444286bd94/volumes" Oct 30 00:24:31.681380 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:31.681334 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:24:31.681599 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:31.681496 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" podUID="3868e018-e83d-4eb5-9aff-842760b30295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:24:31.763184 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:31.763154 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7587-9ad097"] Oct 30 00:24:31.763952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:31.763925 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1584ba56-ffef-49e1-8edd-399ca7b7f366" containerName="istio-proxy" Oct 30 00:24:31.763952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:31.763949 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1584ba56-ffef-49e1-8edd-399ca7b7f366" containerName="istio-proxy" Oct 30 00:24:31.764179 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:31.764167 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1584ba56-ffef-49e1-8edd-399ca7b7f366" containerName="istio-proxy" Oct 30 00:24:31.764223 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:31.764190 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1584ba56-ffef-49e1-8edd-399ca7b7f366" containerName="istio-proxy" Oct 30 00:24:31.771934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:31.771898 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7587-9ad097" Oct 30 00:24:31.786608 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:31.786581 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7587-9ad097"] Oct 30 00:24:31.789588 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:31.789412 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hr9mw\"" Oct 30 00:24:31.862038 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:31.861994 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w72zr\" (UniqueName: \"kubernetes.io/projected/09d2645d-2436-4050-b6e2-c2b3c6f5035b-kube-api-access-w72zr\") pod \"test-curl-7587-9ad097\" (UID: \"09d2645d-2436-4050-b6e2-c2b3c6f5035b\") " pod="opendatahub/test-curl-7587-9ad097" Oct 30 00:24:31.964324 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:31.964238 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w72zr\" (UniqueName: \"kubernetes.io/projected/09d2645d-2436-4050-b6e2-c2b3c6f5035b-kube-api-access-w72zr\") pod \"test-curl-7587-9ad097\" (UID: \"09d2645d-2436-4050-b6e2-c2b3c6f5035b\") " pod="opendatahub/test-curl-7587-9ad097" Oct 30 00:24:31.984952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:31.984899 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w72zr\" (UniqueName: \"kubernetes.io/projected/09d2645d-2436-4050-b6e2-c2b3c6f5035b-kube-api-access-w72zr\") pod \"test-curl-7587-9ad097\" (UID: \"09d2645d-2436-4050-b6e2-c2b3c6f5035b\") " pod="opendatahub/test-curl-7587-9ad097" Oct 30 00:24:32.096152 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:32.096120 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7587-9ad097" Oct 30 00:24:32.344886 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:32.344855 5001 generic.go:356] "Generic (PLEG): container finished" podID="3868e018-e83d-4eb5-9aff-842760b30295" containerID="a8e31ad8114053cd4b4a790aa3a7450c47ae87188ac54c5dce8602bab07e312b" exitCode=0 Oct 30 00:24:32.345185 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:32.344953 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" event={"ID":"3868e018-e83d-4eb5-9aff-842760b30295","Type":"ContainerDied","Data":"a8e31ad8114053cd4b4a790aa3a7450c47ae87188ac54c5dce8602bab07e312b"} Oct 30 00:24:32.345538 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:32.345521 5001 scope.go:122] "RemoveContainer" containerID="a8e31ad8114053cd4b4a790aa3a7450c47ae87188ac54c5dce8602bab07e312b" Oct 30 00:24:32.347310 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:32.347276 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:24:32.347414 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:32.347366 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:24:32.347478 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:32.347418 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:24:33.356885 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:33.356852 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" event={"ID":"3868e018-e83d-4eb5-9aff-842760b30295","Type":"ContainerStarted","Data":"e32e20f75276058c14000dcb4d807d66a65154813c35ecf95df0794e7429c10e"} Oct 30 00:24:33.551018 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:33.550981 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:24:33.551209 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:33.551188 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 00:24:33.551260 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:33.551232 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" podUID="3868e018-e83d-4eb5-9aff-842760b30295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 00:24:34.044894 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:34.044829 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34506eaf-rlg8l"] Oct 30 00:24:34.052433 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:34.052404 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34506eaf-rlg8l"] Oct 30 00:24:34.551613 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:34.551578 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 00:24:34.552089 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:34.551634 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" podUID="3868e018-e83d-4eb5-9aff-842760b30295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 00:24:34.884534 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:34.884488 5001 log.go:32] "RunPodSandbox from runtime service failed" err=< Oct 30 00:24:34.884534 ip-10-0-143-120 kubenswrapper[5001]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_test-curl-7587-9ad097_opendatahub_09d2645d-2436-4050-b6e2-c2b3c6f5035b_0(b1abe7b81831a04a8e889b0249179faa773039a77f9fc96d4f281945cb6e1250): error adding pod opendatahub_test-curl-7587-9ad097 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"b1abe7b81831a04a8e889b0249179faa773039a77f9fc96d4f281945cb6e1250" Netns:"/var/run/netns/75459544-2abb-4a56-8a97-4dcc7e5d004b" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=opendatahub;K8S_POD_NAME=test-curl-7587-9ad097;K8S_POD_INFRA_CONTAINER_ID=b1abe7b81831a04a8e889b0249179faa773039a77f9fc96d4f281945cb6e1250;K8S_POD_UID=09d2645d-2436-4050-b6e2-c2b3c6f5035b" Path:"" ERRORED: error configuring pod [opendatahub/test-curl-7587-9ad097] networking: Multus: [opendatahub/test-curl-7587-9ad097/09d2645d-2436-4050-b6e2-c2b3c6f5035b]: error waiting for pod: Get "https://a883217843c784754a9c94a5ff95cefd-0e67250544583792.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/opendatahub/pods/test-curl-7587-9ad097?timeout=1m0s": context deadline exceeded Oct 30 00:24:34.884534 ip-10-0-143-120 kubenswrapper[5001]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 00:24:34.884534 ip-10-0-143-120 kubenswrapper[5001]: > Oct 30 00:24:34.885083 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:34.884558 5001 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Oct 30 00:24:34.885083 ip-10-0-143-120 kubenswrapper[5001]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_test-curl-7587-9ad097_opendatahub_09d2645d-2436-4050-b6e2-c2b3c6f5035b_0(b1abe7b81831a04a8e889b0249179faa773039a77f9fc96d4f281945cb6e1250): error adding pod opendatahub_test-curl-7587-9ad097 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"b1abe7b81831a04a8e889b0249179faa773039a77f9fc96d4f281945cb6e1250" Netns:"/var/run/netns/75459544-2abb-4a56-8a97-4dcc7e5d004b" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=opendatahub;K8S_POD_NAME=test-curl-7587-9ad097;K8S_POD_INFRA_CONTAINER_ID=b1abe7b81831a04a8e889b0249179faa773039a77f9fc96d4f281945cb6e1250;K8S_POD_UID=09d2645d-2436-4050-b6e2-c2b3c6f5035b" Path:"" ERRORED: error configuring pod [opendatahub/test-curl-7587-9ad097] networking: Multus: [opendatahub/test-curl-7587-9ad097/09d2645d-2436-4050-b6e2-c2b3c6f5035b]: error waiting for pod: Get "https://a883217843c784754a9c94a5ff95cefd-0e67250544583792.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/opendatahub/pods/test-curl-7587-9ad097?timeout=1m0s": context deadline exceeded Oct 30 00:24:34.885083 ip-10-0-143-120 kubenswrapper[5001]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 00:24:34.885083 ip-10-0-143-120 kubenswrapper[5001]: > pod="opendatahub/test-curl-7587-9ad097" Oct 30 00:24:34.885083 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:34.884660 5001 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=< Oct 30 00:24:34.885083 ip-10-0-143-120 kubenswrapper[5001]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_test-curl-7587-9ad097_opendatahub_09d2645d-2436-4050-b6e2-c2b3c6f5035b_0(b1abe7b81831a04a8e889b0249179faa773039a77f9fc96d4f281945cb6e1250): error adding pod opendatahub_test-curl-7587-9ad097 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"b1abe7b81831a04a8e889b0249179faa773039a77f9fc96d4f281945cb6e1250" Netns:"/var/run/netns/75459544-2abb-4a56-8a97-4dcc7e5d004b" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=opendatahub;K8S_POD_NAME=test-curl-7587-9ad097;K8S_POD_INFRA_CONTAINER_ID=b1abe7b81831a04a8e889b0249179faa773039a77f9fc96d4f281945cb6e1250;K8S_POD_UID=09d2645d-2436-4050-b6e2-c2b3c6f5035b" Path:"" ERRORED: error configuring pod [opendatahub/test-curl-7587-9ad097] networking: Multus: [opendatahub/test-curl-7587-9ad097/09d2645d-2436-4050-b6e2-c2b3c6f5035b]: error waiting for pod: Get "https://a883217843c784754a9c94a5ff95cefd-0e67250544583792.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/opendatahub/pods/test-curl-7587-9ad097?timeout=1m0s": context deadline exceeded Oct 30 00:24:34.885083 ip-10-0-143-120 kubenswrapper[5001]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 00:24:34.885083 ip-10-0-143-120 kubenswrapper[5001]: > pod="opendatahub/test-curl-7587-9ad097" Oct 30 00:24:34.885083 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:34.884835 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"test-curl-7587-9ad097_opendatahub(09d2645d-2436-4050-b6e2-c2b3c6f5035b)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"test-curl-7587-9ad097_opendatahub(09d2645d-2436-4050-b6e2-c2b3c6f5035b)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_test-curl-7587-9ad097_opendatahub_09d2645d-2436-4050-b6e2-c2b3c6f5035b_0(b1abe7b81831a04a8e889b0249179faa773039a77f9fc96d4f281945cb6e1250): error adding pod opendatahub_test-curl-7587-9ad097 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"b1abe7b81831a04a8e889b0249179faa773039a77f9fc96d4f281945cb6e1250\\\" Netns:\\\"/var/run/netns/75459544-2abb-4a56-8a97-4dcc7e5d004b\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=opendatahub;K8S_POD_NAME=test-curl-7587-9ad097;K8S_POD_INFRA_CONTAINER_ID=b1abe7b81831a04a8e889b0249179faa773039a77f9fc96d4f281945cb6e1250;K8S_POD_UID=09d2645d-2436-4050-b6e2-c2b3c6f5035b\\\" Path:\\\"\\\" ERRORED: error configuring pod [opendatahub/test-curl-7587-9ad097] networking: Multus: [opendatahub/test-curl-7587-9ad097/09d2645d-2436-4050-b6e2-c2b3c6f5035b]: error waiting for pod: Get \\\"https://a883217843c784754a9c94a5ff95cefd-0e67250544583792.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/opendatahub/pods/test-curl-7587-9ad097?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="opendatahub/test-curl-7587-9ad097" podUID="09d2645d-2436-4050-b6e2-c2b3c6f5035b" Oct 30 00:24:34.946568 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:34.946530 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="273644c7-3e29-4731-81a9-fc1c0b7c4a4a" path="/var/lib/kubelet/pods/273644c7-3e29-4731-81a9-fc1c0b7c4a4a/volumes" Oct 30 00:24:35.375579 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:35.375549 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7587-9ad097" Oct 30 00:24:35.375891 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:35.375878 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7587-9ad097" Oct 30 00:24:35.551642 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:35.551606 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 00:24:35.552092 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:35.551656 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" podUID="3868e018-e83d-4eb5-9aff-842760b30295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 00:24:36.551562 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:36.551527 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 00:24:36.551748 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:36.551582 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" podUID="3868e018-e83d-4eb5-9aff-842760b30295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 00:24:37.551723 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:37.551688 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 00:24:37.551902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:37.551738 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" podUID="3868e018-e83d-4eb5-9aff-842760b30295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 00:24:38.050311 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:38.044237 5001 log.go:32] "RunPodSandbox from runtime service failed" err=< Oct 30 00:24:38.050311 ip-10-0-143-120 kubenswrapper[5001]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_test-curl-7587-9ad097_opendatahub_09d2645d-2436-4050-b6e2-c2b3c6f5035b_0(5b565114fa7263c89ce7654bdd1665325c1a83de49bd1dea89d0cc3d94d42841): error adding pod opendatahub_test-curl-7587-9ad097 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"5b565114fa7263c89ce7654bdd1665325c1a83de49bd1dea89d0cc3d94d42841" Netns:"/var/run/netns/fa63c1e7-2cb5-4ffc-aa85-0f26e1e893f6" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=opendatahub;K8S_POD_NAME=test-curl-7587-9ad097;K8S_POD_INFRA_CONTAINER_ID=5b565114fa7263c89ce7654bdd1665325c1a83de49bd1dea89d0cc3d94d42841;K8S_POD_UID=09d2645d-2436-4050-b6e2-c2b3c6f5035b" Path:"" ERRORED: error configuring pod [opendatahub/test-curl-7587-9ad097] networking: Multus: [opendatahub/test-curl-7587-9ad097/09d2645d-2436-4050-b6e2-c2b3c6f5035b]: error waiting for pod: Get "https://a883217843c784754a9c94a5ff95cefd-0e67250544583792.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/opendatahub/pods/test-curl-7587-9ad097?timeout=1m0s": context deadline exceeded Oct 30 00:24:38.050311 ip-10-0-143-120 kubenswrapper[5001]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 00:24:38.050311 ip-10-0-143-120 kubenswrapper[5001]: > Oct 30 00:24:38.050311 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:38.044296 5001 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Oct 30 00:24:38.050311 ip-10-0-143-120 kubenswrapper[5001]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_test-curl-7587-9ad097_opendatahub_09d2645d-2436-4050-b6e2-c2b3c6f5035b_0(5b565114fa7263c89ce7654bdd1665325c1a83de49bd1dea89d0cc3d94d42841): error adding pod opendatahub_test-curl-7587-9ad097 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"5b565114fa7263c89ce7654bdd1665325c1a83de49bd1dea89d0cc3d94d42841" Netns:"/var/run/netns/fa63c1e7-2cb5-4ffc-aa85-0f26e1e893f6" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=opendatahub;K8S_POD_NAME=test-curl-7587-9ad097;K8S_POD_INFRA_CONTAINER_ID=5b565114fa7263c89ce7654bdd1665325c1a83de49bd1dea89d0cc3d94d42841;K8S_POD_UID=09d2645d-2436-4050-b6e2-c2b3c6f5035b" Path:"" ERRORED: error configuring pod [opendatahub/test-curl-7587-9ad097] networking: Multus: [opendatahub/test-curl-7587-9ad097/09d2645d-2436-4050-b6e2-c2b3c6f5035b]: error waiting for pod: Get "https://a883217843c784754a9c94a5ff95cefd-0e67250544583792.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/opendatahub/pods/test-curl-7587-9ad097?timeout=1m0s": context deadline exceeded Oct 30 00:24:38.050311 ip-10-0-143-120 kubenswrapper[5001]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 00:24:38.050311 ip-10-0-143-120 kubenswrapper[5001]: > pod="opendatahub/test-curl-7587-9ad097" Oct 30 00:24:38.050311 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:38.044321 5001 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=< Oct 30 00:24:38.050311 ip-10-0-143-120 kubenswrapper[5001]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_test-curl-7587-9ad097_opendatahub_09d2645d-2436-4050-b6e2-c2b3c6f5035b_0(5b565114fa7263c89ce7654bdd1665325c1a83de49bd1dea89d0cc3d94d42841): error adding pod opendatahub_test-curl-7587-9ad097 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"5b565114fa7263c89ce7654bdd1665325c1a83de49bd1dea89d0cc3d94d42841" Netns:"/var/run/netns/fa63c1e7-2cb5-4ffc-aa85-0f26e1e893f6" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=opendatahub;K8S_POD_NAME=test-curl-7587-9ad097;K8S_POD_INFRA_CONTAINER_ID=5b565114fa7263c89ce7654bdd1665325c1a83de49bd1dea89d0cc3d94d42841;K8S_POD_UID=09d2645d-2436-4050-b6e2-c2b3c6f5035b" Path:"" ERRORED: error configuring pod [opendatahub/test-curl-7587-9ad097] networking: Multus: [opendatahub/test-curl-7587-9ad097/09d2645d-2436-4050-b6e2-c2b3c6f5035b]: error waiting for pod: Get "https://a883217843c784754a9c94a5ff95cefd-0e67250544583792.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/opendatahub/pods/test-curl-7587-9ad097?timeout=1m0s": context deadline exceeded Oct 30 00:24:38.050311 ip-10-0-143-120 kubenswrapper[5001]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 00:24:38.050311 ip-10-0-143-120 kubenswrapper[5001]: > pod="opendatahub/test-curl-7587-9ad097" Oct 30 00:24:38.050311 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:38.044399 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"test-curl-7587-9ad097_opendatahub(09d2645d-2436-4050-b6e2-c2b3c6f5035b)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"test-curl-7587-9ad097_opendatahub(09d2645d-2436-4050-b6e2-c2b3c6f5035b)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_test-curl-7587-9ad097_opendatahub_09d2645d-2436-4050-b6e2-c2b3c6f5035b_0(5b565114fa7263c89ce7654bdd1665325c1a83de49bd1dea89d0cc3d94d42841): error adding pod opendatahub_test-curl-7587-9ad097 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"5b565114fa7263c89ce7654bdd1665325c1a83de49bd1dea89d0cc3d94d42841\\\" Netns:\\\"/var/run/netns/fa63c1e7-2cb5-4ffc-aa85-0f26e1e893f6\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=opendatahub;K8S_POD_NAME=test-curl-7587-9ad097;K8S_POD_INFRA_CONTAINER_ID=5b565114fa7263c89ce7654bdd1665325c1a83de49bd1dea89d0cc3d94d42841;K8S_POD_UID=09d2645d-2436-4050-b6e2-c2b3c6f5035b\\\" Path:\\\"\\\" ERRORED: error configuring pod [opendatahub/test-curl-7587-9ad097] networking: Multus: [opendatahub/test-curl-7587-9ad097/09d2645d-2436-4050-b6e2-c2b3c6f5035b]: error waiting for pod: Get \\\"https://a883217843c784754a9c94a5ff95cefd-0e67250544583792.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/opendatahub/pods/test-curl-7587-9ad097?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="opendatahub/test-curl-7587-9ad097" podUID="09d2645d-2436-4050-b6e2-c2b3c6f5035b" Oct 30 00:24:38.551087 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:38.551055 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 00:24:38.551278 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:38.551115 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" podUID="3868e018-e83d-4eb5-9aff-842760b30295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 00:24:39.615933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:39.577161 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:24:39.615933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:39.578866 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:24:39.615933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:39.584893 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:24:43.753940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:43.749957 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-84dc9c97ff-6tsx5"] Oct 30 00:24:43.753940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:43.751190 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1584ba56-ffef-49e1-8edd-399ca7b7f366" containerName="istio-proxy" Oct 30 00:24:43.753940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:43.751212 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1584ba56-ffef-49e1-8edd-399ca7b7f366" containerName="istio-proxy" Oct 30 00:24:43.778949 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:43.774586 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-84dc9c97ff-6tsx5" Oct 30 00:24:43.837669 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:43.837315 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6a55e1\"" Oct 30 00:24:43.837669 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:43.837481 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6a55e1-dockercfg-xdghp\"" Oct 30 00:24:43.918222 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:43.918190 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" Oct 30 00:24:43.955243 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:43.930459 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" Oct 30 00:24:43.955243 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:43.918529 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78" Oct 30 00:24:43.955243 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:43.931078 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78" Oct 30 00:24:43.955243 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:43.919533 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2nkw\" (UniqueName: \"kubernetes.io/projected/de1c480c-7dbc-47b0-bdff-d9accfbb0944-kube-api-access-q2nkw\") pod \"payload-pre-processing-e2e-ipp-b-6a55e1-84dc9c97ff-6tsx5\" (UID: \"de1c480c-7dbc-47b0-bdff-d9accfbb0944\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-84dc9c97ff-6tsx5" Oct 30 00:24:43.955243 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:43.931708 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de1c480c-7dbc-47b0-bdff-d9accfbb0944-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6a55e1-84dc9c97ff-6tsx5\" (UID: \"de1c480c-7dbc-47b0-bdff-d9accfbb0944\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-84dc9c97ff-6tsx5" Oct 30 00:24:44.017520 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:44.015324 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-84dc9c97ff-6tsx5"] Oct 30 00:24:44.034568 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:44.034510 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-7b5c564646-k975p"] Oct 30 00:24:44.047650 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:44.047623 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-7b5c564646-k975p" Oct 30 00:24:44.056647 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:44.056440 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2nkw\" (UniqueName: \"kubernetes.io/projected/de1c480c-7dbc-47b0-bdff-d9accfbb0944-kube-api-access-q2nkw\") pod \"payload-pre-processing-e2e-ipp-b-6a55e1-84dc9c97ff-6tsx5\" (UID: \"de1c480c-7dbc-47b0-bdff-d9accfbb0944\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-84dc9c97ff-6tsx5" Oct 30 00:24:44.056868 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:44.056790 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de1c480c-7dbc-47b0-bdff-d9accfbb0944-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6a55e1-84dc9c97ff-6tsx5\" (UID: \"de1c480c-7dbc-47b0-bdff-d9accfbb0944\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-84dc9c97ff-6tsx5" Oct 30 00:24:44.063226 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:44.063181 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-7b5c564646-k975p"] Oct 30 00:24:44.116962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:44.116183 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de1c480c-7dbc-47b0-bdff-d9accfbb0944-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6a55e1-84dc9c97ff-6tsx5\" (UID: \"de1c480c-7dbc-47b0-bdff-d9accfbb0944\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-84dc9c97ff-6tsx5" Oct 30 00:24:44.200424 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:44.200384 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/badf6bc7-f9da-431f-8007-9f7b53cec3a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6a55e1-7b5c564646-k975p\" (UID: \"badf6bc7-f9da-431f-8007-9f7b53cec3a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-7b5c564646-k975p" Oct 30 00:24:44.200986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:44.200756 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r6pf\" (UniqueName: \"kubernetes.io/projected/badf6bc7-f9da-431f-8007-9f7b53cec3a1-kube-api-access-5r6pf\") pod \"payload-processing-e2e-ipp-b-6a55e1-7b5c564646-k975p\" (UID: \"badf6bc7-f9da-431f-8007-9f7b53cec3a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-7b5c564646-k975p" Oct 30 00:24:44.223360 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:44.223333 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2nkw\" (UniqueName: \"kubernetes.io/projected/de1c480c-7dbc-47b0-bdff-d9accfbb0944-kube-api-access-q2nkw\") pod \"payload-pre-processing-e2e-ipp-b-6a55e1-84dc9c97ff-6tsx5\" (UID: \"de1c480c-7dbc-47b0-bdff-d9accfbb0944\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-84dc9c97ff-6tsx5" Oct 30 00:24:44.302035 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:44.301960 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/badf6bc7-f9da-431f-8007-9f7b53cec3a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6a55e1-7b5c564646-k975p\" (UID: \"badf6bc7-f9da-431f-8007-9f7b53cec3a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-7b5c564646-k975p" Oct 30 00:24:44.302236 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:44.302222 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r6pf\" (UniqueName: \"kubernetes.io/projected/badf6bc7-f9da-431f-8007-9f7b53cec3a1-kube-api-access-5r6pf\") pod \"payload-processing-e2e-ipp-b-6a55e1-7b5c564646-k975p\" (UID: \"badf6bc7-f9da-431f-8007-9f7b53cec3a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-7b5c564646-k975p" Oct 30 00:24:44.302680 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:44.302653 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/badf6bc7-f9da-431f-8007-9f7b53cec3a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6a55e1-7b5c564646-k975p\" (UID: \"badf6bc7-f9da-431f-8007-9f7b53cec3a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-7b5c564646-k975p" Oct 30 00:24:44.324151 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:44.324121 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r6pf\" (UniqueName: \"kubernetes.io/projected/badf6bc7-f9da-431f-8007-9f7b53cec3a1-kube-api-access-5r6pf\") pod \"payload-processing-e2e-ipp-b-6a55e1-7b5c564646-k975p\" (UID: \"badf6bc7-f9da-431f-8007-9f7b53cec3a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-7b5c564646-k975p" Oct 30 00:24:44.510998 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:44.508626 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-84dc9c97ff-6tsx5" Oct 30 00:24:44.563874 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:44.563813 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-7b5c564646-k975p" Oct 30 00:24:45.230123 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:45.230096 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8gb5_9fa688d8-29ca-4c1f-926a-f1a4099e6b44/console-operator/2.log" Oct 30 00:24:45.232222 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:45.232195 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8gb5_9fa688d8-29ca-4c1f-926a-f1a4099e6b44/console-operator/2.log" Oct 30 00:24:46.060439 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:46.060067 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96027e-76ddd5685d-v4sqk"] Oct 30 00:24:46.113247 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:46.107263 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96027e-76ddd5685d-v4sqk"] Oct 30 00:24:46.113247 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:46.107391 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-76ddd5685d-v4sqk" Oct 30 00:24:46.134057 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:46.118486 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f08b449-d648r"] Oct 30 00:24:46.134057 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:46.121404 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-96027e-dockercfg-m7vjn\"" Oct 30 00:24:46.134057 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:46.123129 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-96027e\"" Oct 30 00:24:46.134057 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:46.125847 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f08b449-d648r"] Oct 30 00:24:46.237866 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:46.237834 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnptb\" (UniqueName: \"kubernetes.io/projected/c6ba9acb-5f2f-4028-9963-fe2201dccffe-kube-api-access-lnptb\") pod \"payload-processing-e2e-ipp-b-96027e-76ddd5685d-v4sqk\" (UID: \"c6ba9acb-5f2f-4028-9963-fe2201dccffe\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-76ddd5685d-v4sqk" Oct 30 00:24:46.238374 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:46.237964 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6ba9acb-5f2f-4028-9963-fe2201dccffe-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-96027e-76ddd5685d-v4sqk\" (UID: \"c6ba9acb-5f2f-4028-9963-fe2201dccffe\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-76ddd5685d-v4sqk" Oct 30 00:24:46.339576 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:46.339472 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnptb\" (UniqueName: \"kubernetes.io/projected/c6ba9acb-5f2f-4028-9963-fe2201dccffe-kube-api-access-lnptb\") pod \"payload-processing-e2e-ipp-b-96027e-76ddd5685d-v4sqk\" (UID: \"c6ba9acb-5f2f-4028-9963-fe2201dccffe\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-76ddd5685d-v4sqk" Oct 30 00:24:46.339576 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:46.339569 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6ba9acb-5f2f-4028-9963-fe2201dccffe-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-96027e-76ddd5685d-v4sqk\" (UID: \"c6ba9acb-5f2f-4028-9963-fe2201dccffe\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-76ddd5685d-v4sqk" Oct 30 00:24:46.340405 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:46.340382 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6ba9acb-5f2f-4028-9963-fe2201dccffe-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-96027e-76ddd5685d-v4sqk\" (UID: \"c6ba9acb-5f2f-4028-9963-fe2201dccffe\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-76ddd5685d-v4sqk" Oct 30 00:24:46.409870 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:46.409838 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnptb\" (UniqueName: \"kubernetes.io/projected/c6ba9acb-5f2f-4028-9963-fe2201dccffe-kube-api-access-lnptb\") pod \"payload-processing-e2e-ipp-b-96027e-76ddd5685d-v4sqk\" (UID: \"c6ba9acb-5f2f-4028-9963-fe2201dccffe\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-76ddd5685d-v4sqk" Oct 30 00:24:46.466311 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:46.466260 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-76ddd5685d-v4sqk" Oct 30 00:24:46.816588 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:46.816556 5001 log.go:32] "RunPodSandbox from runtime service failed" err=< Oct 30 00:24:46.816588 ip-10-0-143-120 kubenswrapper[5001]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk_odh-ai-gateway-infra_1da01660-8624-4707-bd34-edba0f103f2f_0(e118f1de11e84e8f06cde09efe1866e8fa9921d48cdf90cb0263168f76ad58db): error adding pod odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"e118f1de11e84e8f06cde09efe1866e8fa9921d48cdf90cb0263168f76ad58db" Netns:"/var/run/netns/f114de04-20a7-4a6b-b1fc-c06d9eb93106" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk;K8S_POD_INFRA_CONTAINER_ID=e118f1de11e84e8f06cde09efe1866e8fa9921d48cdf90cb0263168f76ad58db;K8S_POD_UID=1da01660-8624-4707-bd34-edba0f103f2f" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk] networking: Multus: [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk/1da01660-8624-4707-bd34-edba0f103f2f]: error waiting for pod: unexpected error when reading response body. Please retry. Original error: context deadline exceeded Oct 30 00:24:46.816588 ip-10-0-143-120 kubenswrapper[5001]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 00:24:46.816588 ip-10-0-143-120 kubenswrapper[5001]: > Oct 30 00:24:46.816758 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:46.816616 5001 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Oct 30 00:24:46.816758 ip-10-0-143-120 kubenswrapper[5001]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk_odh-ai-gateway-infra_1da01660-8624-4707-bd34-edba0f103f2f_0(e118f1de11e84e8f06cde09efe1866e8fa9921d48cdf90cb0263168f76ad58db): error adding pod odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"e118f1de11e84e8f06cde09efe1866e8fa9921d48cdf90cb0263168f76ad58db" Netns:"/var/run/netns/f114de04-20a7-4a6b-b1fc-c06d9eb93106" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk;K8S_POD_INFRA_CONTAINER_ID=e118f1de11e84e8f06cde09efe1866e8fa9921d48cdf90cb0263168f76ad58db;K8S_POD_UID=1da01660-8624-4707-bd34-edba0f103f2f" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk] networking: Multus: [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk/1da01660-8624-4707-bd34-edba0f103f2f]: error waiting for pod: unexpected error when reading response body. Please retry. Original error: context deadline exceeded Oct 30 00:24:46.816758 ip-10-0-143-120 kubenswrapper[5001]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 00:24:46.816758 ip-10-0-143-120 kubenswrapper[5001]: > pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" Oct 30 00:24:46.816758 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:46.816636 5001 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=< Oct 30 00:24:46.816758 ip-10-0-143-120 kubenswrapper[5001]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk_odh-ai-gateway-infra_1da01660-8624-4707-bd34-edba0f103f2f_0(e118f1de11e84e8f06cde09efe1866e8fa9921d48cdf90cb0263168f76ad58db): error adding pod odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"e118f1de11e84e8f06cde09efe1866e8fa9921d48cdf90cb0263168f76ad58db" Netns:"/var/run/netns/f114de04-20a7-4a6b-b1fc-c06d9eb93106" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk;K8S_POD_INFRA_CONTAINER_ID=e118f1de11e84e8f06cde09efe1866e8fa9921d48cdf90cb0263168f76ad58db;K8S_POD_UID=1da01660-8624-4707-bd34-edba0f103f2f" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk] networking: Multus: [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk/1da01660-8624-4707-bd34-edba0f103f2f]: error waiting for pod: unexpected error when reading response body. Please retry. Original error: context deadline exceeded Oct 30 00:24:46.816758 ip-10-0-143-120 kubenswrapper[5001]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 00:24:46.816758 ip-10-0-143-120 kubenswrapper[5001]: > pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" Oct 30 00:24:46.816758 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:46.816714 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk_odh-ai-gateway-infra(1da01660-8624-4707-bd34-edba0f103f2f)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk_odh-ai-gateway-infra(1da01660-8624-4707-bd34-edba0f103f2f)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk_odh-ai-gateway-infra_1da01660-8624-4707-bd34-edba0f103f2f_0(e118f1de11e84e8f06cde09efe1866e8fa9921d48cdf90cb0263168f76ad58db): error adding pod odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"e118f1de11e84e8f06cde09efe1866e8fa9921d48cdf90cb0263168f76ad58db\\\" Netns:\\\"/var/run/netns/f114de04-20a7-4a6b-b1fc-c06d9eb93106\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk;K8S_POD_INFRA_CONTAINER_ID=e118f1de11e84e8f06cde09efe1866e8fa9921d48cdf90cb0263168f76ad58db;K8S_POD_UID=1da01660-8624-4707-bd34-edba0f103f2f\\\" Path:\\\"\\\" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk] networking: Multus: [odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk/1da01660-8624-4707-bd34-edba0f103f2f]: error waiting for pod: unexpected error when reading response body. Please retry. Original error: context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" podUID="1da01660-8624-4707-bd34-edba0f103f2f" Oct 30 00:24:46.937190 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:46.937150 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="824446e0-437d-4fd3-902c-5afdfacda635" path="/var/lib/kubelet/pods/824446e0-437d-4fd3-902c-5afdfacda635/volumes" Oct 30 00:24:47.046888 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:24:47.046860 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccde7ce6_b81a_4564_a8f2_15669279f5a9.slice/crio-593550b0e555680ad269f668748d18c509e77771a554efb0efcbe21be44b6f64 WatchSource:0}: Error finding container 593550b0e555680ad269f668748d18c509e77771a554efb0efcbe21be44b6f64: Status 404 returned error can't find the container with id 593550b0e555680ad269f668748d18c509e77771a554efb0efcbe21be44b6f64 Oct 30 00:24:47.329105 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:47.329075 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:24:47.444056 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:24:47.444027 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbadf6bc7_f9da_431f_8007_9f7b53cec3a1.slice/crio-aaf4021b852b597cfec0b2b8b5dadeed199788b72ce19afd2a6be3466876e3bf WatchSource:0}: Error finding container aaf4021b852b597cfec0b2b8b5dadeed199788b72ce19afd2a6be3466876e3bf: Status 404 returned error can't find the container with id aaf4021b852b597cfec0b2b8b5dadeed199788b72ce19afd2a6be3466876e3bf Oct 30 00:24:47.504369 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:47.504344 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l"] Oct 30 00:24:47.504818 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:47.504775 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:24:47.504895 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:47.504862 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:24:47.504985 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:47.504923 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:24:47.581425 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:47.581338 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78" event={"ID":"ccde7ce6-b81a-4564-a8f2-15669279f5a9","Type":"ContainerStarted","Data":"f66c1d659908c8cb0f066ea0d1e15c8f1c92e933289d5bb02a290bde64491b2c"} Oct 30 00:24:47.581425 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:47.581409 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78" event={"ID":"ccde7ce6-b81a-4564-a8f2-15669279f5a9","Type":"ContainerStarted","Data":"593550b0e555680ad269f668748d18c509e77771a554efb0efcbe21be44b6f64"} Oct 30 00:24:47.583272 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:47.583244 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-84dc9c97ff-6tsx5" event={"ID":"de1c480c-7dbc-47b0-bdff-d9accfbb0944","Type":"ContainerStarted","Data":"ba9a6745f35e9b36e6ad069a8ed0b7c0b28dedde8d719dd28e990afd7bb5f2ed"} Oct 30 00:24:47.583430 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:47.583296 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-84dc9c97ff-6tsx5" event={"ID":"de1c480c-7dbc-47b0-bdff-d9accfbb0944","Type":"ContainerStarted","Data":"3d24754752fc618067144152fb654fabc31e77d060ab6160b7ec9e7cbd10dc92"} Oct 30 00:24:47.583430 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:47.583329 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-84dc9c97ff-6tsx5" Oct 30 00:24:47.584887 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:47.584862 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" event={"ID":"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb","Type":"ContainerStarted","Data":"1f85e49783fe866d42ff8785dae084b430799d61291cc4bd688ab5b8701462f4"} Oct 30 00:24:47.587076 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:47.587051 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-7b5c564646-k975p" event={"ID":"badf6bc7-f9da-431f-8007-9f7b53cec3a1","Type":"ContainerStarted","Data":"460666f4942a66c66b96715d4dde6993258d701de591a843b16bd524dbf2d3a0"} Oct 30 00:24:47.587176 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:47.587098 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-7b5c564646-k975p" event={"ID":"badf6bc7-f9da-431f-8007-9f7b53cec3a1","Type":"ContainerStarted","Data":"aaf4021b852b597cfec0b2b8b5dadeed199788b72ce19afd2a6be3466876e3bf"} Oct 30 00:24:47.587225 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:47.587208 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-7b5c564646-k975p" Oct 30 00:24:47.627865 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:47.627801 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78" podStartSLOduration=27.627782207 podStartE2EDuration="27.627782207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:24:47.622097858 +0000 UTC m=+1202.827599039" watchObservedRunningTime="2026-10-30 00:24:47.627782207 +0000 UTC m=+1202.833283389" Oct 30 00:24:47.663721 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:47.663297 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-7b5c564646-k975p" podStartSLOduration=4.663278396 podStartE2EDuration="4.663278396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:24:47.657209756 +0000 UTC m=+1202.862710941" watchObservedRunningTime="2026-10-30 00:24:47.663278396 +0000 UTC m=+1202.868779581" Oct 30 00:24:47.696418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:47.696357 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-84dc9c97ff-6tsx5" podStartSLOduration=4.696341504 podStartE2EDuration="4.696341504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:24:47.692423913 +0000 UTC m=+1202.897925114" watchObservedRunningTime="2026-10-30 00:24:47.696341504 +0000 UTC m=+1202.901842685" Oct 30 00:24:48.254001 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:48.253958 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96027e-76ddd5685d-v4sqk"] Oct 30 00:24:48.268875 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:24:48.268844 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ba9acb_5f2f_4028_9963_fe2201dccffe.slice/crio-f2359c52538e174d0ff1087fff2e5ff1e2af7f3f247612a992b08c8af308d2c0 WatchSource:0}: Error finding container f2359c52538e174d0ff1087fff2e5ff1e2af7f3f247612a992b08c8af308d2c0: Status 404 returned error can't find the container with id f2359c52538e174d0ff1087fff2e5ff1e2af7f3f247612a992b08c8af308d2c0 Oct 30 00:24:48.599579 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:48.599480 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-76ddd5685d-v4sqk" event={"ID":"c6ba9acb-5f2f-4028-9963-fe2201dccffe","Type":"ContainerStarted","Data":"382793f6756a34a19463fb58d36ff50b5582b757a1fe27543ce80e7fe3c67dcb"} Oct 30 00:24:48.599579 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:48.599548 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-76ddd5685d-v4sqk" event={"ID":"c6ba9acb-5f2f-4028-9963-fe2201dccffe","Type":"ContainerStarted","Data":"f2359c52538e174d0ff1087fff2e5ff1e2af7f3f247612a992b08c8af308d2c0"} Oct 30 00:24:48.600109 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:48.599626 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-76ddd5685d-v4sqk" Oct 30 00:24:48.601367 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:48.601346 5001 generic.go:356] "Generic (PLEG): container finished" podID="ccde7ce6-b81a-4564-a8f2-15669279f5a9" containerID="f66c1d659908c8cb0f066ea0d1e15c8f1c92e933289d5bb02a290bde64491b2c" exitCode=0 Oct 30 00:24:48.601508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:48.601415 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78" event={"ID":"ccde7ce6-b81a-4564-a8f2-15669279f5a9","Type":"ContainerDied","Data":"f66c1d659908c8cb0f066ea0d1e15c8f1c92e933289d5bb02a290bde64491b2c"} Oct 30 00:24:48.603303 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:48.603282 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" event={"ID":"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb","Type":"ContainerStarted","Data":"7735f81688c824ce4c03de2ab668e577ef7e34cae26d89e81364b4b573038fcc"} Oct 30 00:24:48.645214 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:48.645149 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-76ddd5685d-v4sqk" podStartSLOduration=3.645135455 podStartE2EDuration="3.645135455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:24:48.641259663 +0000 UTC m=+1203.846760844" watchObservedRunningTime="2026-10-30 00:24:48.645135455 +0000 UTC m=+1203.850636635" Oct 30 00:24:48.719676 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:48.719620 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" podStartSLOduration=48.719597537 podStartE2EDuration="48.719597537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:24:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:24:48.714953766 +0000 UTC m=+1203.920454948" watchObservedRunningTime="2026-10-30 00:24:48.719597537 +0000 UTC m=+1203.925098782" Oct 30 00:24:48.742290 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:48.742258 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:24:48.748211 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:48.748187 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:24:49.615117 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:49.615083 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:24:49.616364 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:49.616345 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:24:49.995932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:49.974147 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78" Oct 30 00:24:50.148506 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:50.148467 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccde7ce6-b81a-4564-a8f2-15669279f5a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccde7ce6-b81a-4564-a8f2-15669279f5a9-maas-api-openshift-ca\") pod \"ccde7ce6-b81a-4564-a8f2-15669279f5a9\" (UID: \"ccde7ce6-b81a-4564-a8f2-15669279f5a9\") " Oct 30 00:24:50.148701 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:50.148619 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccde7ce6-b81a-4564-a8f2-15669279f5a9-kube-api-access-b6bs7\" (UniqueName: \"kubernetes.io/projected/ccde7ce6-b81a-4564-a8f2-15669279f5a9-kube-api-access-b6bs7\") pod \"ccde7ce6-b81a-4564-a8f2-15669279f5a9\" (UID: \"ccde7ce6-b81a-4564-a8f2-15669279f5a9\") " Oct 30 00:24:50.148701 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:50.148679 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccde7ce6-b81a-4564-a8f2-15669279f5a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccde7ce6-b81a-4564-a8f2-15669279f5a9-maas-api-xks-ca\") pod \"ccde7ce6-b81a-4564-a8f2-15669279f5a9\" (UID: \"ccde7ce6-b81a-4564-a8f2-15669279f5a9\") " Oct 30 00:24:50.148904 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:50.148874 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccde7ce6-b81a-4564-a8f2-15669279f5a9-maas-api-openshift-ca" pod "ccde7ce6-b81a-4564-a8f2-15669279f5a9" (UID: "ccde7ce6-b81a-4564-a8f2-15669279f5a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:24:50.149385 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:50.149365 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccde7ce6-b81a-4564-a8f2-15669279f5a9-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:24:50.150836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:50.150805 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccde7ce6-b81a-4564-a8f2-15669279f5a9-maas-api-xks-ca" pod "ccde7ce6-b81a-4564-a8f2-15669279f5a9" (UID: "ccde7ce6-b81a-4564-a8f2-15669279f5a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:24:50.150975 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:50.150887 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccde7ce6-b81a-4564-a8f2-15669279f5a9-kube-api-access-b6bs7" pod "ccde7ce6-b81a-4564-a8f2-15669279f5a9" (UID: "ccde7ce6-b81a-4564-a8f2-15669279f5a9"). InnerVolumeSpecName "kube-api-access-b6bs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:24:50.251000 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:50.250872 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6bs7\" (UniqueName: \"kubernetes.io/projected/ccde7ce6-b81a-4564-a8f2-15669279f5a9-kube-api-access-b6bs7\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:24:50.251000 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:50.250903 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccde7ce6-b81a-4564-a8f2-15669279f5a9-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:24:50.431492 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:50.431455 5001 scope.go:122] "RemoveContainer" containerID="dbf10a479bf4c142cdaa352311d5a3ee182038547c48600b4ddc4d4e0964fcc7" Oct 30 00:24:50.461190 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:50.461113 5001 scope.go:122] "RemoveContainer" containerID="085fbd7be1965654c617b3218427b4b1439dd4769ff5a2b9682b36c3f69931b2" Oct 30 00:24:50.475018 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:50.474993 5001 scope.go:122] "RemoveContainer" containerID="e5956a44b8d39d61cd4e1d58cf00a2591f4f6ce5056fabfd31a6594245e20163" Oct 30 00:24:51.080075 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:51.077579 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78" Oct 30 00:24:51.198968 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:51.198082 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78" event={"ID":"ccde7ce6-b81a-4564-a8f2-15669279f5a9","Type":"ContainerDied","Data":"593550b0e555680ad269f668748d18c509e77771a554efb0efcbe21be44b6f64"} Oct 30 00:24:51.198968 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:51.198159 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="593550b0e555680ad269f668748d18c509e77771a554efb0efcbe21be44b6f64" Oct 30 00:24:51.910971 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:51.910937 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7587-9ad097" Oct 30 00:24:51.911401 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:51.911386 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7587-9ad097" Oct 30 00:24:52.230271 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:52.230243 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7587-9ad097"] Oct 30 00:24:52.231000 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:24:52.230971 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09d2645d_2436_4050_b6e2_c2b3c6f5035b.slice/crio-a9e2d6a0393206eb2f0c896289c6f3865e96f1c367c0774596b717252d71d6d2 WatchSource:0}: Error finding container a9e2d6a0393206eb2f0c896289c6f3865e96f1c367c0774596b717252d71d6d2: Status 404 returned error can't find the container with id a9e2d6a0393206eb2f0c896289c6f3865e96f1c367c0774596b717252d71d6d2 Oct 30 00:24:52.887399 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:52.887359 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-6b65cf7999-jkkfd"] Oct 30 00:24:52.888249 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:52.888229 5001 cpu_manager.go:414] "containerMap: removing container" podUID="ccde7ce6-b81a-4564-a8f2-15669279f5a9" containerName="revoke-keys" Oct 30 00:24:52.888249 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:52.888249 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccde7ce6-b81a-4564-a8f2-15669279f5a9" containerName="revoke-keys" Oct 30 00:24:52.888377 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:52.888361 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccde7ce6-b81a-4564-a8f2-15669279f5a9" containerName="revoke-keys" Oct 30 00:24:52.892536 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:52.892516 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-6b65cf7999-jkkfd" Oct 30 00:24:52.914192 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:52.914155 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f8397b-dockercfg-2sx76\"" Oct 30 00:24:52.929054 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:52.929022 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-6b65cf7999-jkkfd"] Oct 30 00:24:53.022935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:52.958481 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f8397b\"" Oct 30 00:24:53.119937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:53.083780 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f91ba282-c730-4f93-ba5b-92a63bad3488-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f8397b-6b65cf7999-jkkfd\" (UID: \"f91ba282-c730-4f93-ba5b-92a63bad3488\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-6b65cf7999-jkkfd" Oct 30 00:24:53.119937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:53.083874 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbdnw\" (UniqueName: \"kubernetes.io/projected/f91ba282-c730-4f93-ba5b-92a63bad3488-kube-api-access-rbdnw\") pod \"payload-pre-processing-e2e-ipp-a-f8397b-6b65cf7999-jkkfd\" (UID: \"f91ba282-c730-4f93-ba5b-92a63bad3488\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-6b65cf7999-jkkfd" Oct 30 00:24:53.215993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:53.210358 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f91ba282-c730-4f93-ba5b-92a63bad3488-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f8397b-6b65cf7999-jkkfd\" (UID: \"f91ba282-c730-4f93-ba5b-92a63bad3488\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-6b65cf7999-jkkfd" Oct 30 00:24:53.215993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:53.210433 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbdnw\" (UniqueName: \"kubernetes.io/projected/f91ba282-c730-4f93-ba5b-92a63bad3488-kube-api-access-rbdnw\") pod \"payload-pre-processing-e2e-ipp-a-f8397b-6b65cf7999-jkkfd\" (UID: \"f91ba282-c730-4f93-ba5b-92a63bad3488\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-6b65cf7999-jkkfd" Oct 30 00:24:53.257969 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:53.256640 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f91ba282-c730-4f93-ba5b-92a63bad3488-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f8397b-6b65cf7999-jkkfd\" (UID: \"f91ba282-c730-4f93-ba5b-92a63bad3488\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-6b65cf7999-jkkfd" Oct 30 00:24:53.290381 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:53.290338 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7587-9ad097" event={"ID":"09d2645d-2436-4050-b6e2-c2b3c6f5035b","Type":"ContainerStarted","Data":"0edd64818c6398dac689c0617632c05f935a9c1f3d25cf3306e6bab87a8ba02c"} Oct 30 00:24:53.290660 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:53.290631 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7587-9ad097" event={"ID":"09d2645d-2436-4050-b6e2-c2b3c6f5035b","Type":"ContainerStarted","Data":"a9e2d6a0393206eb2f0c896289c6f3865e96f1c367c0774596b717252d71d6d2"} Oct 30 00:24:53.322292 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:53.322262 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbdnw\" (UniqueName: \"kubernetes.io/projected/f91ba282-c730-4f93-ba5b-92a63bad3488-kube-api-access-rbdnw\") pod \"payload-pre-processing-e2e-ipp-a-f8397b-6b65cf7999-jkkfd\" (UID: \"f91ba282-c730-4f93-ba5b-92a63bad3488\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-6b65cf7999-jkkfd" Oct 30 00:24:53.350610 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:53.350551 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7587-9ad097" podStartSLOduration=22.350523604 podStartE2EDuration="22.350523604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:24:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:24:53.34159081 +0000 UTC m=+1208.547091985" watchObservedRunningTime="2026-10-30 00:24:53.350523604 +0000 UTC m=+1208.556024788" Oct 30 00:24:53.521370 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:53.521324 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-6b65cf7999-jkkfd" Oct 30 00:24:53.995814 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:24:53.995781 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf91ba282_c730_4f93_ba5b_92a63bad3488.slice/crio-04253af86b85c37a22077493f7a7733a30ca53cf26f384ea3c1b692e790460ae WatchSource:0}: Error finding container 04253af86b85c37a22077493f7a7733a30ca53cf26f384ea3c1b692e790460ae: Status 404 returned error can't find the container with id 04253af86b85c37a22077493f7a7733a30ca53cf26f384ea3c1b692e790460ae Oct 30 00:24:53.997214 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:53.997193 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-6b65cf7999-jkkfd"] Oct 30 00:24:54.301212 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:54.301177 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-6b65cf7999-jkkfd" event={"ID":"f91ba282-c730-4f93-ba5b-92a63bad3488","Type":"ContainerStarted","Data":"904f1d651fadd6bd68979e5892f4c6eec717598a47c684f82b6c24b7428d0ec4"} Oct 30 00:24:54.301607 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:54.301263 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-6b65cf7999-jkkfd" event={"ID":"f91ba282-c730-4f93-ba5b-92a63bad3488","Type":"ContainerStarted","Data":"04253af86b85c37a22077493f7a7733a30ca53cf26f384ea3c1b692e790460ae"} Oct 30 00:24:54.332010 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:54.331978 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7587-9ad097"] Oct 30 00:24:54.337389 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:54.337359 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7587-9ad097"] Oct 30 00:24:54.342411 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:54.342356 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-6b65cf7999-jkkfd" podStartSLOduration=2.342338414 podStartE2EDuration="2.342338414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:24:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:24:54.335497947 +0000 UTC m=+1209.540999129" watchObservedRunningTime="2026-10-30 00:24:54.342338414 +0000 UTC m=+1209.547839596" Oct 30 00:24:54.704218 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:54.704134 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-hw97x"] Oct 30 00:24:54.704987 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:54.704971 5001 cpu_manager.go:414] "containerMap: removing container" podUID="09d2645d-2436-4050-b6e2-c2b3c6f5035b" containerName="test-curl-7587-9ad097" Oct 30 00:24:54.705048 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:54.704989 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09d2645d-2436-4050-b6e2-c2b3c6f5035b" containerName="test-curl-7587-9ad097" Oct 30 00:24:54.705148 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:54.705139 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="09d2645d-2436-4050-b6e2-c2b3c6f5035b" containerName="test-curl-7587-9ad097" Oct 30 00:24:54.709319 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:54.709303 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-hw97x" Oct 30 00:24:54.731713 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:54.731679 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-hw97x"] Oct 30 00:24:54.787084 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:54.787052 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc4lq\" (UniqueName: \"kubernetes.io/projected/a83e6d25-723d-48ed-9679-565e3953acce-kube-api-access-vc4lq\") pod \"maas-api-revoke-keys-e2e-mt-0847e8c0-hw97x\" (UID: \"a83e6d25-723d-48ed-9679-565e3953acce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-hw97x" Oct 30 00:24:54.787250 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:54.787117 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a83e6d25-723d-48ed-9679-565e3953acce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-0847e8c0-hw97x\" (UID: \"a83e6d25-723d-48ed-9679-565e3953acce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-hw97x" Oct 30 00:24:54.787376 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:54.787358 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a83e6d25-723d-48ed-9679-565e3953acce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-0847e8c0-hw97x\" (UID: \"a83e6d25-723d-48ed-9679-565e3953acce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-hw97x" Oct 30 00:24:54.891283 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:54.891239 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a83e6d25-723d-48ed-9679-565e3953acce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-0847e8c0-hw97x\" (UID: \"a83e6d25-723d-48ed-9679-565e3953acce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-hw97x" Oct 30 00:24:54.891764 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:54.891735 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc4lq\" (UniqueName: \"kubernetes.io/projected/a83e6d25-723d-48ed-9679-565e3953acce-kube-api-access-vc4lq\") pod \"maas-api-revoke-keys-e2e-mt-0847e8c0-hw97x\" (UID: \"a83e6d25-723d-48ed-9679-565e3953acce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-hw97x" Oct 30 00:24:54.891975 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:54.891958 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a83e6d25-723d-48ed-9679-565e3953acce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-0847e8c0-hw97x\" (UID: \"a83e6d25-723d-48ed-9679-565e3953acce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-hw97x" Oct 30 00:24:54.893234 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:54.893182 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a83e6d25-723d-48ed-9679-565e3953acce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-0847e8c0-hw97x\" (UID: \"a83e6d25-723d-48ed-9679-565e3953acce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-hw97x" Oct 30 00:24:54.948991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:54.948388 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a83e6d25-723d-48ed-9679-565e3953acce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-0847e8c0-hw97x\" (UID: \"a83e6d25-723d-48ed-9679-565e3953acce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-hw97x" Oct 30 00:24:54.949971 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:54.949945 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc4lq\" (UniqueName: \"kubernetes.io/projected/a83e6d25-723d-48ed-9679-565e3953acce-kube-api-access-vc4lq\") pod \"maas-api-revoke-keys-e2e-mt-0847e8c0-hw97x\" (UID: \"a83e6d25-723d-48ed-9679-565e3953acce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-hw97x" Oct 30 00:24:55.024494 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:55.024454 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-hw97x" Oct 30 00:24:55.310588 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:55.310477 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7587-9ad097" podUID="09d2645d-2436-4050-b6e2-c2b3c6f5035b" containerName="test-curl-7587-9ad097" containerID="cri-o://0edd64818c6398dac689c0617632c05f935a9c1f3d25cf3306e6bab87a8ba02c" gracePeriod=2 Oct 30 00:24:55.311086 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:55.310701 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-6b65cf7999-jkkfd" Oct 30 00:24:55.334475 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:24:55.334442 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83e6d25_723d_48ed_9679_565e3953acce.slice/crio-7bc0ec540a728274ee267aaa3684ad1da07e51cc5975791a3a642e8ae26fd5d2 WatchSource:0}: Error finding container 7bc0ec540a728274ee267aaa3684ad1da07e51cc5975791a3a642e8ae26fd5d2: Status 404 returned error can't find the container with id 7bc0ec540a728274ee267aaa3684ad1da07e51cc5975791a3a642e8ae26fd5d2 Oct 30 00:24:55.335717 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:55.335691 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-hw97x"] Oct 30 00:24:56.321436 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:56.321401 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-hw97x" event={"ID":"a83e6d25-723d-48ed-9679-565e3953acce","Type":"ContainerStarted","Data":"a90f65f8608f50128e6b206a0d82dd1089527785e617d4122065befef5172f1b"} Oct 30 00:24:56.321832 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:56.321463 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-hw97x" event={"ID":"a83e6d25-723d-48ed-9679-565e3953acce","Type":"ContainerStarted","Data":"7bc0ec540a728274ee267aaa3684ad1da07e51cc5975791a3a642e8ae26fd5d2"} Oct 30 00:24:56.360978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:56.360904 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-hw97x" podStartSLOduration=2.360889831 podStartE2EDuration="2.360889831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:24:56.356609647 +0000 UTC m=+1211.562110827" watchObservedRunningTime="2026-10-30 00:24:56.360889831 +0000 UTC m=+1211.566391012" Oct 30 00:24:56.388298 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:56.388267 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-7f5fd784c9-pjrvt"] Oct 30 00:24:56.393101 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:56.393082 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-7f5fd784c9-pjrvt" Oct 30 00:24:56.408658 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:56.408630 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3fb3ad\"" Oct 30 00:24:56.408798 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:56.408784 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3fb3ad-dockercfg-r5758\"" Oct 30 00:24:56.416164 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:56.416143 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-7f5fd784c9-pjrvt"] Oct 30 00:24:56.425289 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:56.425258 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259e6702-4256-4bbe-b2e0-86b4d2071f66-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3fb3ad-7f5fd784c9-pjrvt\" (UID: \"259e6702-4256-4bbe-b2e0-86b4d2071f66\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-7f5fd784c9-pjrvt" Oct 30 00:24:56.425651 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:56.425628 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrg85\" (UniqueName: \"kubernetes.io/projected/259e6702-4256-4bbe-b2e0-86b4d2071f66-kube-api-access-rrg85\") pod \"payload-pre-processing-e2e-ipp-a-3fb3ad-7f5fd784c9-pjrvt\" (UID: \"259e6702-4256-4bbe-b2e0-86b4d2071f66\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-7f5fd784c9-pjrvt" Oct 30 00:24:56.582983 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:56.571137 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrg85\" (UniqueName: \"kubernetes.io/projected/259e6702-4256-4bbe-b2e0-86b4d2071f66-kube-api-access-rrg85\") pod \"payload-pre-processing-e2e-ipp-a-3fb3ad-7f5fd784c9-pjrvt\" (UID: \"259e6702-4256-4bbe-b2e0-86b4d2071f66\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-7f5fd784c9-pjrvt" Oct 30 00:24:56.582983 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:56.571357 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259e6702-4256-4bbe-b2e0-86b4d2071f66-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3fb3ad-7f5fd784c9-pjrvt\" (UID: \"259e6702-4256-4bbe-b2e0-86b4d2071f66\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-7f5fd784c9-pjrvt" Oct 30 00:24:56.626934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:56.614986 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259e6702-4256-4bbe-b2e0-86b4d2071f66-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3fb3ad-7f5fd784c9-pjrvt\" (UID: \"259e6702-4256-4bbe-b2e0-86b4d2071f66\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-7f5fd784c9-pjrvt" Oct 30 00:24:56.750935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:56.750014 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrg85\" (UniqueName: \"kubernetes.io/projected/259e6702-4256-4bbe-b2e0-86b4d2071f66-kube-api-access-rrg85\") pod \"payload-pre-processing-e2e-ipp-a-3fb3ad-7f5fd784c9-pjrvt\" (UID: \"259e6702-4256-4bbe-b2e0-86b4d2071f66\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-7f5fd784c9-pjrvt" Oct 30 00:24:57.011318 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:57.011189 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-7f5fd784c9-pjrvt" Oct 30 00:24:57.328441 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:24:57.328412 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod259e6702_4256_4bbe_b2e0_86b4d2071f66.slice/crio-272a6bef0307e8c6b1581bf27be8106c398e781bb295f920e58bf6835f4809c9 WatchSource:0}: Error finding container 272a6bef0307e8c6b1581bf27be8106c398e781bb295f920e58bf6835f4809c9: Status 404 returned error can't find the container with id 272a6bef0307e8c6b1581bf27be8106c398e781bb295f920e58bf6835f4809c9 Oct 30 00:24:57.328837 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:57.328815 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-7f5fd784c9-pjrvt"] Oct 30 00:24:57.334005 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:57.333980 5001 generic.go:356] "Generic (PLEG): container finished" podID="a83e6d25-723d-48ed-9679-565e3953acce" containerID="a90f65f8608f50128e6b206a0d82dd1089527785e617d4122065befef5172f1b" exitCode=0 Oct 30 00:24:57.334105 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:57.334067 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-hw97x" event={"ID":"a83e6d25-723d-48ed-9679-565e3953acce","Type":"ContainerDied","Data":"a90f65f8608f50128e6b206a0d82dd1089527785e617d4122065befef5172f1b"} Oct 30 00:24:57.523445 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:57.523417 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7587-9ad097" Oct 30 00:24:57.716093 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:57.716012 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09d2645d-2436-4050-b6e2-c2b3c6f5035b-kube-api-access-w72zr\" (UniqueName: \"kubernetes.io/projected/09d2645d-2436-4050-b6e2-c2b3c6f5035b-kube-api-access-w72zr\") pod \"09d2645d-2436-4050-b6e2-c2b3c6f5035b\" (UID: \"09d2645d-2436-4050-b6e2-c2b3c6f5035b\") " Oct 30 00:24:57.718060 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:57.718031 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d2645d-2436-4050-b6e2-c2b3c6f5035b-kube-api-access-w72zr" pod "09d2645d-2436-4050-b6e2-c2b3c6f5035b" (UID: "09d2645d-2436-4050-b6e2-c2b3c6f5035b"). InnerVolumeSpecName "kube-api-access-w72zr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:24:57.817818 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:57.817776 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w72zr\" (UniqueName: \"kubernetes.io/projected/09d2645d-2436-4050-b6e2-c2b3c6f5035b-kube-api-access-w72zr\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:24:57.910113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:57.910084 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" Oct 30 00:24:57.910485 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:57.910472 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" Oct 30 00:24:58.345211 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:58.345180 5001 generic.go:356] "Generic (PLEG): container finished" podID="09d2645d-2436-4050-b6e2-c2b3c6f5035b" containerID="0edd64818c6398dac689c0617632c05f935a9c1f3d25cf3306e6bab87a8ba02c" exitCode=137 Oct 30 00:24:58.345683 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:58.345223 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7587-9ad097" Oct 30 00:24:58.345683 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:58.345279 5001 scope.go:122] "RemoveContainer" containerID="0edd64818c6398dac689c0617632c05f935a9c1f3d25cf3306e6bab87a8ba02c" Oct 30 00:24:58.347339 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:58.347311 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-7f5fd784c9-pjrvt" event={"ID":"259e6702-4256-4bbe-b2e0-86b4d2071f66","Type":"ContainerStarted","Data":"1a049553580afe928257072be796eaeb02b25aa307040fbfc686923d646557a8"} Oct 30 00:24:58.347465 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:58.347369 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-7f5fd784c9-pjrvt" event={"ID":"259e6702-4256-4bbe-b2e0-86b4d2071f66","Type":"ContainerStarted","Data":"272a6bef0307e8c6b1581bf27be8106c398e781bb295f920e58bf6835f4809c9"} Oct 30 00:24:58.347465 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:58.347410 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-7f5fd784c9-pjrvt" Oct 30 00:24:58.359071 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:58.359053 5001 scope.go:122] "RemoveContainer" containerID="0edd64818c6398dac689c0617632c05f935a9c1f3d25cf3306e6bab87a8ba02c" Oct 30 00:24:58.359377 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:58.359348 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0edd64818c6398dac689c0617632c05f935a9c1f3d25cf3306e6bab87a8ba02c\": container with ID starting with 0edd64818c6398dac689c0617632c05f935a9c1f3d25cf3306e6bab87a8ba02c not found: ID does not exist" containerID="0edd64818c6398dac689c0617632c05f935a9c1f3d25cf3306e6bab87a8ba02c" Oct 30 00:24:58.359431 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:58.359386 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0edd64818c6398dac689c0617632c05f935a9c1f3d25cf3306e6bab87a8ba02c"} err="failed to get container status \"0edd64818c6398dac689c0617632c05f935a9c1f3d25cf3306e6bab87a8ba02c\": rpc error: code = NotFound desc = could not find container \"0edd64818c6398dac689c0617632c05f935a9c1f3d25cf3306e6bab87a8ba02c\": container with ID starting with 0edd64818c6398dac689c0617632c05f935a9c1f3d25cf3306e6bab87a8ba02c not found: ID does not exist" Oct 30 00:24:58.386423 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:58.386364 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-7f5fd784c9-pjrvt" podStartSLOduration=2.386344825 podStartE2EDuration="2.386344825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:24:58.381411483 +0000 UTC m=+1213.586912668" watchObservedRunningTime="2026-10-30 00:24:58.386344825 +0000 UTC m=+1213.591846007" Oct 30 00:24:58.392604 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:58.392576 5001 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7587-9ad097\" is forbidden: User \"system:node:ip-10-0-143-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-120.ec2.internal' and this object" podUID="09d2645d-2436-4050-b6e2-c2b3c6f5035b" pod="opendatahub/test-curl-7587-9ad097" Oct 30 00:24:58.432196 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:58.432167 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk"] Oct 30 00:24:58.440483 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:24:58.434981 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da01660_8624_4707_bd34_edba0f103f2f.slice/crio-d20011711fd68c8a352c2d683f5db2920b3133826fd7cd55748834919e16e998 WatchSource:0}: Error finding container d20011711fd68c8a352c2d683f5db2920b3133826fd7cd55748834919e16e998: Status 404 returned error can't find the container with id d20011711fd68c8a352c2d683f5db2920b3133826fd7cd55748834919e16e998 Oct 30 00:24:58.633462 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:58.633062 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-84dc9c97ff-6tsx5" Oct 30 00:24:58.633462 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:58.633149 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-7b5c564646-k975p" Oct 30 00:24:58.662393 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:58.662360 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-hw97x" Oct 30 00:24:58.725282 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:58.725240 5001 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7587-9ad097\" is forbidden: User \"system:node:ip-10-0-143-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-120.ec2.internal' and this object" podUID="09d2645d-2436-4050-b6e2-c2b3c6f5035b" pod="opendatahub/test-curl-7587-9ad097" Oct 30 00:24:58.736400 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:24:58.736362 5001 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7587-9ad097\" is forbidden: User \"system:node:ip-10-0-143-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-120.ec2.internal' and this object" podUID="09d2645d-2436-4050-b6e2-c2b3c6f5035b" pod="opendatahub/test-curl-7587-9ad097" Oct 30 00:24:58.763010 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:58.762977 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a83e6d25-723d-48ed-9679-565e3953acce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a83e6d25-723d-48ed-9679-565e3953acce-maas-api-xks-ca\") pod \"a83e6d25-723d-48ed-9679-565e3953acce\" (UID: \"a83e6d25-723d-48ed-9679-565e3953acce\") " Oct 30 00:24:58.763171 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:58.763080 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a83e6d25-723d-48ed-9679-565e3953acce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a83e6d25-723d-48ed-9679-565e3953acce-maas-api-openshift-ca\") pod \"a83e6d25-723d-48ed-9679-565e3953acce\" (UID: \"a83e6d25-723d-48ed-9679-565e3953acce\") " Oct 30 00:24:58.763171 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:58.763132 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a83e6d25-723d-48ed-9679-565e3953acce-kube-api-access-vc4lq\" (UniqueName: \"kubernetes.io/projected/a83e6d25-723d-48ed-9679-565e3953acce-kube-api-access-vc4lq\") pod \"a83e6d25-723d-48ed-9679-565e3953acce\" (UID: \"a83e6d25-723d-48ed-9679-565e3953acce\") " Oct 30 00:24:58.763523 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:58.763491 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83e6d25-723d-48ed-9679-565e3953acce-maas-api-openshift-ca" pod "a83e6d25-723d-48ed-9679-565e3953acce" (UID: "a83e6d25-723d-48ed-9679-565e3953acce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:24:58.764319 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:58.764126 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a83e6d25-723d-48ed-9679-565e3953acce-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:24:58.765272 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:58.765250 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a83e6d25-723d-48ed-9679-565e3953acce-maas-api-xks-ca" pod "a83e6d25-723d-48ed-9679-565e3953acce" (UID: "a83e6d25-723d-48ed-9679-565e3953acce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:24:58.765556 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:58.765532 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83e6d25-723d-48ed-9679-565e3953acce-kube-api-access-vc4lq" pod "a83e6d25-723d-48ed-9679-565e3953acce" (UID: "a83e6d25-723d-48ed-9679-565e3953acce"). InnerVolumeSpecName "kube-api-access-vc4lq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:24:58.866193 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:58.866109 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a83e6d25-723d-48ed-9679-565e3953acce-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:24:58.866193 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:58.866136 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc4lq\" (UniqueName: \"kubernetes.io/projected/a83e6d25-723d-48ed-9679-565e3953acce-kube-api-access-vc4lq\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:24:58.922300 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:58.922267 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09d2645d-2436-4050-b6e2-c2b3c6f5035b" path="/var/lib/kubelet/pods/09d2645d-2436-4050-b6e2-c2b3c6f5035b/volumes" Oct 30 00:24:59.358642 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:59.358614 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-hw97x" Oct 30 00:24:59.359093 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:59.358643 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-hw97x" event={"ID":"a83e6d25-723d-48ed-9679-565e3953acce","Type":"ContainerDied","Data":"7bc0ec540a728274ee267aaa3684ad1da07e51cc5975791a3a642e8ae26fd5d2"} Oct 30 00:24:59.359093 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:59.358696 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bc0ec540a728274ee267aaa3684ad1da07e51cc5975791a3a642e8ae26fd5d2" Oct 30 00:24:59.362061 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:59.362025 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" event={"ID":"1da01660-8624-4707-bd34-edba0f103f2f","Type":"ContainerStarted","Data":"80cd54c2ee4c671480b0df1d9c2f1af5dae91887f808bec5a6f1e40f3b4e9211"} Oct 30 00:24:59.362220 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:59.362074 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" event={"ID":"1da01660-8624-4707-bd34-edba0f103f2f","Type":"ContainerStarted","Data":"d20011711fd68c8a352c2d683f5db2920b3133826fd7cd55748834919e16e998"} Oct 30 00:24:59.416682 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:59.412057 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" podStartSLOduration=51.412039373 podStartE2EDuration="51.412039373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:24:59.40834184 +0000 UTC m=+1214.613843021" watchObservedRunningTime="2026-10-30 00:24:59.412039373 +0000 UTC m=+1214.617540559" Oct 30 00:24:59.617697 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:24:59.617623 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-76ddd5685d-v4sqk" Oct 30 00:25:00.386687 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:00.386654 5001 generic.go:356] "Generic (PLEG): container finished" podID="1da01660-8624-4707-bd34-edba0f103f2f" containerID="80cd54c2ee4c671480b0df1d9c2f1af5dae91887f808bec5a6f1e40f3b4e9211" exitCode=0 Oct 30 00:25:00.387120 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:00.386739 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" event={"ID":"1da01660-8624-4707-bd34-edba0f103f2f","Type":"ContainerDied","Data":"80cd54c2ee4c671480b0df1d9c2f1af5dae91887f808bec5a6f1e40f3b4e9211"} Oct 30 00:25:01.451944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:01.435725 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c34304-glt92"] Oct 30 00:25:01.451944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:01.437886 5001 cpu_manager.go:414] "containerMap: removing container" podUID="a83e6d25-723d-48ed-9679-565e3953acce" containerName="revoke-keys" Oct 30 00:25:01.451944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:01.437933 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83e6d25-723d-48ed-9679-565e3953acce" containerName="revoke-keys" Oct 30 00:25:01.458298 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:01.458263 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83e6d25-723d-48ed-9679-565e3953acce" containerName="revoke-keys" Oct 30 00:25:01.479952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:01.477629 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c34304-glt92" Oct 30 00:25:01.551108 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:01.528239 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442d9e02-2a7b-4573-918d-9e0c39bbed23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c34304-glt92\" (UID: \"442d9e02-2a7b-4573-918d-9e0c39bbed23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c34304-glt92" Oct 30 00:25:01.580937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:01.562086 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxlm2\" (UniqueName: \"kubernetes.io/projected/442d9e02-2a7b-4573-918d-9e0c39bbed23-kube-api-access-fxlm2\") pod \"maas-api-revoke-keys-e2e-ipp-a-c34304-glt92\" (UID: \"442d9e02-2a7b-4573-918d-9e0c39bbed23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c34304-glt92" Oct 30 00:25:01.580937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:01.562210 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442d9e02-2a7b-4573-918d-9e0c39bbed23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c34304-glt92\" (UID: \"442d9e02-2a7b-4573-918d-9e0c39bbed23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c34304-glt92" Oct 30 00:25:01.587051 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:01.586973 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c34304-glt92"] Oct 30 00:25:01.665801 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:01.665711 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442d9e02-2a7b-4573-918d-9e0c39bbed23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c34304-glt92\" (UID: \"442d9e02-2a7b-4573-918d-9e0c39bbed23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c34304-glt92" Oct 30 00:25:01.724289 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:01.710224 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxlm2\" (UniqueName: \"kubernetes.io/projected/442d9e02-2a7b-4573-918d-9e0c39bbed23-kube-api-access-fxlm2\") pod \"maas-api-revoke-keys-e2e-ipp-a-c34304-glt92\" (UID: \"442d9e02-2a7b-4573-918d-9e0c39bbed23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c34304-glt92" Oct 30 00:25:01.798831 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:01.786033 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442d9e02-2a7b-4573-918d-9e0c39bbed23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c34304-glt92\" (UID: \"442d9e02-2a7b-4573-918d-9e0c39bbed23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c34304-glt92" Oct 30 00:25:01.798831 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:01.787201 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442d9e02-2a7b-4573-918d-9e0c39bbed23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c34304-glt92\" (UID: \"442d9e02-2a7b-4573-918d-9e0c39bbed23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c34304-glt92" Oct 30 00:25:01.825634 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:01.824310 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f52ea1-x2dh9"] Oct 30 00:25:01.837377 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:01.837321 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442d9e02-2a7b-4573-918d-9e0c39bbed23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c34304-glt92\" (UID: \"442d9e02-2a7b-4573-918d-9e0c39bbed23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c34304-glt92" Oct 30 00:25:01.838210 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:01.838168 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f52ea1-x2dh9" Oct 30 00:25:01.845179 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:01.845147 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f52ea1-x2dh9"] Oct 30 00:25:01.891939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:01.872769 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxlm2\" (UniqueName: \"kubernetes.io/projected/442d9e02-2a7b-4573-918d-9e0c39bbed23-kube-api-access-fxlm2\") pod \"maas-api-revoke-keys-e2e-ipp-a-c34304-glt92\" (UID: \"442d9e02-2a7b-4573-918d-9e0c39bbed23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c34304-glt92" Oct 30 00:25:01.891939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:01.884295 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c34304-glt92" Oct 30 00:25:01.902363 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:01.894402 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68d5a825-45b1-45ec-9d34-1b141f6e6daa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f52ea1-x2dh9\" (UID: \"68d5a825-45b1-45ec-9d34-1b141f6e6daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f52ea1-x2dh9" Oct 30 00:25:01.902363 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:01.894508 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz5lh\" (UniqueName: \"kubernetes.io/projected/68d5a825-45b1-45ec-9d34-1b141f6e6daa-kube-api-access-mz5lh\") pod \"maas-api-revoke-keys-e2e-ipp-a-f52ea1-x2dh9\" (UID: \"68d5a825-45b1-45ec-9d34-1b141f6e6daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f52ea1-x2dh9" Oct 30 00:25:01.902363 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:01.894727 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68d5a825-45b1-45ec-9d34-1b141f6e6daa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f52ea1-x2dh9\" (UID: \"68d5a825-45b1-45ec-9d34-1b141f6e6daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f52ea1-x2dh9" Oct 30 00:25:01.999995 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:01.997891 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68d5a825-45b1-45ec-9d34-1b141f6e6daa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f52ea1-x2dh9\" (UID: \"68d5a825-45b1-45ec-9d34-1b141f6e6daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f52ea1-x2dh9" Oct 30 00:25:01.999995 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:01.998187 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68d5a825-45b1-45ec-9d34-1b141f6e6daa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f52ea1-x2dh9\" (UID: \"68d5a825-45b1-45ec-9d34-1b141f6e6daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f52ea1-x2dh9" Oct 30 00:25:01.999995 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:01.998412 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz5lh\" (UniqueName: \"kubernetes.io/projected/68d5a825-45b1-45ec-9d34-1b141f6e6daa-kube-api-access-mz5lh\") pod \"maas-api-revoke-keys-e2e-ipp-a-f52ea1-x2dh9\" (UID: \"68d5a825-45b1-45ec-9d34-1b141f6e6daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f52ea1-x2dh9" Oct 30 00:25:01.999995 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:01.999634 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68d5a825-45b1-45ec-9d34-1b141f6e6daa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f52ea1-x2dh9\" (UID: \"68d5a825-45b1-45ec-9d34-1b141f6e6daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f52ea1-x2dh9" Oct 30 00:25:02.001872 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:02.001844 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68d5a825-45b1-45ec-9d34-1b141f6e6daa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f52ea1-x2dh9\" (UID: \"68d5a825-45b1-45ec-9d34-1b141f6e6daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f52ea1-x2dh9" Oct 30 00:25:02.048027 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:02.045946 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz5lh\" (UniqueName: \"kubernetes.io/projected/68d5a825-45b1-45ec-9d34-1b141f6e6daa-kube-api-access-mz5lh\") pod \"maas-api-revoke-keys-e2e-ipp-a-f52ea1-x2dh9\" (UID: \"68d5a825-45b1-45ec-9d34-1b141f6e6daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f52ea1-x2dh9" Oct 30 00:25:02.160781 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:02.160622 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" Oct 30 00:25:02.257263 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:02.257184 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f52ea1-x2dh9" Oct 30 00:25:02.303866 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:02.303830 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1da01660-8624-4707-bd34-edba0f103f2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1da01660-8624-4707-bd34-edba0f103f2f-maas-api-xks-ca\") pod \"1da01660-8624-4707-bd34-edba0f103f2f\" (UID: \"1da01660-8624-4707-bd34-edba0f103f2f\") " Oct 30 00:25:02.304118 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:02.304053 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1da01660-8624-4707-bd34-edba0f103f2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1da01660-8624-4707-bd34-edba0f103f2f-maas-api-openshift-ca\") pod \"1da01660-8624-4707-bd34-edba0f103f2f\" (UID: \"1da01660-8624-4707-bd34-edba0f103f2f\") " Oct 30 00:25:02.304351 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:02.304181 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1da01660-8624-4707-bd34-edba0f103f2f-kube-api-access-6p64q\" (UniqueName: \"kubernetes.io/projected/1da01660-8624-4707-bd34-edba0f103f2f-kube-api-access-6p64q\") pod \"1da01660-8624-4707-bd34-edba0f103f2f\" (UID: \"1da01660-8624-4707-bd34-edba0f103f2f\") " Oct 30 00:25:02.306005 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:02.305949 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1da01660-8624-4707-bd34-edba0f103f2f-maas-api-openshift-ca" pod "1da01660-8624-4707-bd34-edba0f103f2f" (UID: "1da01660-8624-4707-bd34-edba0f103f2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:25:02.330956 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:25:02.320883 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod442d9e02_2a7b_4573_918d_9e0c39bbed23.slice/crio-d6e2b97bf6d20d9da7bbb6ff57cdc85ec5fe1179739e222d3e1949388c124099 WatchSource:0}: Error finding container d6e2b97bf6d20d9da7bbb6ff57cdc85ec5fe1179739e222d3e1949388c124099: Status 404 returned error can't find the container with id d6e2b97bf6d20d9da7bbb6ff57cdc85ec5fe1179739e222d3e1949388c124099 Oct 30 00:25:02.330956 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:02.321026 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1da01660-8624-4707-bd34-edba0f103f2f-maas-api-xks-ca" pod "1da01660-8624-4707-bd34-edba0f103f2f" (UID: "1da01660-8624-4707-bd34-edba0f103f2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:25:02.330956 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:02.321151 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1da01660-8624-4707-bd34-edba0f103f2f-kube-api-access-6p64q" pod "1da01660-8624-4707-bd34-edba0f103f2f" (UID: "1da01660-8624-4707-bd34-edba0f103f2f"). InnerVolumeSpecName "kube-api-access-6p64q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:02.343013 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:02.342973 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c34304-glt92"] Oct 30 00:25:02.407557 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:02.407520 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1da01660-8624-4707-bd34-edba0f103f2f-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:02.407557 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:02.407559 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p64q\" (UniqueName: \"kubernetes.io/projected/1da01660-8624-4707-bd34-edba0f103f2f-kube-api-access-6p64q\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:02.407776 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:02.407576 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1da01660-8624-4707-bd34-edba0f103f2f-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:02.450255 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:02.450198 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c34304-glt92" event={"ID":"442d9e02-2a7b-4573-918d-9e0c39bbed23","Type":"ContainerStarted","Data":"d6e2b97bf6d20d9da7bbb6ff57cdc85ec5fe1179739e222d3e1949388c124099"} Oct 30 00:25:02.452326 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:02.452297 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" event={"ID":"1da01660-8624-4707-bd34-edba0f103f2f","Type":"ContainerDied","Data":"d20011711fd68c8a352c2d683f5db2920b3133826fd7cd55748834919e16e998"} Oct 30 00:25:02.452326 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:02.452319 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk" Oct 30 00:25:02.452777 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:02.452343 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d20011711fd68c8a352c2d683f5db2920b3133826fd7cd55748834919e16e998" Oct 30 00:25:02.652056 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:02.651941 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f52ea1-x2dh9"] Oct 30 00:25:03.463847 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:03.463807 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f52ea1-x2dh9" event={"ID":"68d5a825-45b1-45ec-9d34-1b141f6e6daa","Type":"ContainerStarted","Data":"951177bcf7f535391505c41b5321337cf90af5467d26ce382a588652cb91e60c"} Oct 30 00:25:03.464411 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:03.463873 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f52ea1-x2dh9" event={"ID":"68d5a825-45b1-45ec-9d34-1b141f6e6daa","Type":"ContainerStarted","Data":"73ca7d2b4b6a79234763c75ee4ecf432e9830e814f44436cc2aeb7ad647c6b1e"} Oct 30 00:25:03.465774 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:03.465738 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c34304-glt92" event={"ID":"442d9e02-2a7b-4573-918d-9e0c39bbed23","Type":"ContainerStarted","Data":"674d726a595c86b83bfc3c06cbf202a45dbcb5c1018260a9bdbf049fb269c85a"} Oct 30 00:25:03.498126 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:03.498079 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f52ea1-x2dh9" podStartSLOduration=2.498063121 podStartE2EDuration="2.498063121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:25:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:25:03.497710194 +0000 UTC m=+1218.703211396" watchObservedRunningTime="2026-10-30 00:25:03.498063121 +0000 UTC m=+1218.703564301" Oct 30 00:25:03.556066 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:03.556009 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c34304-glt92" podStartSLOduration=2.555990624 podStartE2EDuration="2.555990624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:25:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:25:03.549542228 +0000 UTC m=+1218.755043420" watchObservedRunningTime="2026-10-30 00:25:03.555990624 +0000 UTC m=+1218.761491806" Oct 30 00:25:04.042934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:04.042848 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe23b4ca-pcwj9"] Oct 30 00:25:04.048903 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:04.048867 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe23b4ca-pcwj9"] Oct 30 00:25:04.477507 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:04.477425 5001 generic.go:356] "Generic (PLEG): container finished" podID="68d5a825-45b1-45ec-9d34-1b141f6e6daa" containerID="951177bcf7f535391505c41b5321337cf90af5467d26ce382a588652cb91e60c" exitCode=0 Oct 30 00:25:04.477904 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:04.477511 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f52ea1-x2dh9" event={"ID":"68d5a825-45b1-45ec-9d34-1b141f6e6daa","Type":"ContainerDied","Data":"951177bcf7f535391505c41b5321337cf90af5467d26ce382a588652cb91e60c"} Oct 30 00:25:04.479472 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:04.479448 5001 generic.go:356] "Generic (PLEG): container finished" podID="442d9e02-2a7b-4573-918d-9e0c39bbed23" containerID="674d726a595c86b83bfc3c06cbf202a45dbcb5c1018260a9bdbf049fb269c85a" exitCode=0 Oct 30 00:25:04.479590 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:04.479531 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c34304-glt92" event={"ID":"442d9e02-2a7b-4573-918d-9e0c39bbed23","Type":"ContainerDied","Data":"674d726a595c86b83bfc3c06cbf202a45dbcb5c1018260a9bdbf049fb269c85a"} Oct 30 00:25:04.944435 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:04.943497 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e917f1ed-a050-40ef-b78e-43f1fbdbf461" path="/var/lib/kubelet/pods/e917f1ed-a050-40ef-b78e-43f1fbdbf461/volumes" Oct 30 00:25:06.166293 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:06.165896 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c34304-glt92" Oct 30 00:25:06.309057 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:06.307647 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/442d9e02-2a7b-4573-918d-9e0c39bbed23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442d9e02-2a7b-4573-918d-9e0c39bbed23-maas-api-xks-ca\") pod \"442d9e02-2a7b-4573-918d-9e0c39bbed23\" (UID: \"442d9e02-2a7b-4573-918d-9e0c39bbed23\") " Oct 30 00:25:06.309057 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:06.307810 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/442d9e02-2a7b-4573-918d-9e0c39bbed23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442d9e02-2a7b-4573-918d-9e0c39bbed23-maas-api-openshift-ca\") pod \"442d9e02-2a7b-4573-918d-9e0c39bbed23\" (UID: \"442d9e02-2a7b-4573-918d-9e0c39bbed23\") " Oct 30 00:25:06.309057 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:06.307961 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/442d9e02-2a7b-4573-918d-9e0c39bbed23-kube-api-access-fxlm2\" (UniqueName: \"kubernetes.io/projected/442d9e02-2a7b-4573-918d-9e0c39bbed23-kube-api-access-fxlm2\") pod \"442d9e02-2a7b-4573-918d-9e0c39bbed23\" (UID: \"442d9e02-2a7b-4573-918d-9e0c39bbed23\") " Oct 30 00:25:06.328442 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:06.325421 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f52ea1-x2dh9" Oct 30 00:25:06.339321 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:06.328729 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/442d9e02-2a7b-4573-918d-9e0c39bbed23-maas-api-openshift-ca" pod "442d9e02-2a7b-4573-918d-9e0c39bbed23" (UID: "442d9e02-2a7b-4573-918d-9e0c39bbed23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:25:06.368929 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:06.366161 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/442d9e02-2a7b-4573-918d-9e0c39bbed23-kube-api-access-fxlm2" pod "442d9e02-2a7b-4573-918d-9e0c39bbed23" (UID: "442d9e02-2a7b-4573-918d-9e0c39bbed23"). InnerVolumeSpecName "kube-api-access-fxlm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:06.394615 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:06.393513 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-6b65cf7999-jkkfd" Oct 30 00:25:06.394928 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:06.394873 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/442d9e02-2a7b-4573-918d-9e0c39bbed23-maas-api-xks-ca" pod "442d9e02-2a7b-4573-918d-9e0c39bbed23" (UID: "442d9e02-2a7b-4573-918d-9e0c39bbed23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:25:06.413410 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:06.413371 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68d5a825-45b1-45ec-9d34-1b141f6e6daa-kube-api-access-mz5lh\" (UniqueName: \"kubernetes.io/projected/68d5a825-45b1-45ec-9d34-1b141f6e6daa-kube-api-access-mz5lh\") pod \"68d5a825-45b1-45ec-9d34-1b141f6e6daa\" (UID: \"68d5a825-45b1-45ec-9d34-1b141f6e6daa\") " Oct 30 00:25:06.413823 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:06.413803 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68d5a825-45b1-45ec-9d34-1b141f6e6daa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68d5a825-45b1-45ec-9d34-1b141f6e6daa-maas-api-openshift-ca\") pod \"68d5a825-45b1-45ec-9d34-1b141f6e6daa\" (UID: \"68d5a825-45b1-45ec-9d34-1b141f6e6daa\") " Oct 30 00:25:06.414031 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:06.414011 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68d5a825-45b1-45ec-9d34-1b141f6e6daa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68d5a825-45b1-45ec-9d34-1b141f6e6daa-maas-api-xks-ca\") pod \"68d5a825-45b1-45ec-9d34-1b141f6e6daa\" (UID: \"68d5a825-45b1-45ec-9d34-1b141f6e6daa\") " Oct 30 00:25:06.415007 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:06.414986 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442d9e02-2a7b-4573-918d-9e0c39bbed23-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:06.415161 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:06.415146 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442d9e02-2a7b-4573-918d-9e0c39bbed23-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:06.415280 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:06.415266 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxlm2\" (UniqueName: \"kubernetes.io/projected/442d9e02-2a7b-4573-918d-9e0c39bbed23-kube-api-access-fxlm2\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:06.445956 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:06.445534 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68d5a825-45b1-45ec-9d34-1b141f6e6daa-maas-api-openshift-ca" pod "68d5a825-45b1-45ec-9d34-1b141f6e6daa" (UID: "68d5a825-45b1-45ec-9d34-1b141f6e6daa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:25:06.472145 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:06.472107 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d5a825-45b1-45ec-9d34-1b141f6e6daa-kube-api-access-mz5lh" pod "68d5a825-45b1-45ec-9d34-1b141f6e6daa" (UID: "68d5a825-45b1-45ec-9d34-1b141f6e6daa"). InnerVolumeSpecName "kube-api-access-mz5lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:06.534899 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:06.534857 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz5lh\" (UniqueName: \"kubernetes.io/projected/68d5a825-45b1-45ec-9d34-1b141f6e6daa-kube-api-access-mz5lh\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:06.535183 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:06.535163 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68d5a825-45b1-45ec-9d34-1b141f6e6daa-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:06.606694 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:06.606599 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68d5a825-45b1-45ec-9d34-1b141f6e6daa-maas-api-xks-ca" pod "68d5a825-45b1-45ec-9d34-1b141f6e6daa" (UID: "68d5a825-45b1-45ec-9d34-1b141f6e6daa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:25:06.665073 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:06.661154 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68d5a825-45b1-45ec-9d34-1b141f6e6daa-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:06.800936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:06.797600 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c34304-glt92" event={"ID":"442d9e02-2a7b-4573-918d-9e0c39bbed23","Type":"ContainerDied","Data":"d6e2b97bf6d20d9da7bbb6ff57cdc85ec5fe1179739e222d3e1949388c124099"} Oct 30 00:25:06.800936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:06.797672 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6e2b97bf6d20d9da7bbb6ff57cdc85ec5fe1179739e222d3e1949388c124099" Oct 30 00:25:06.800936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:06.797748 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c34304-glt92" Oct 30 00:25:06.808151 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:06.808099 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f52ea1-x2dh9" event={"ID":"68d5a825-45b1-45ec-9d34-1b141f6e6daa","Type":"ContainerDied","Data":"73ca7d2b4b6a79234763c75ee4ecf432e9830e814f44436cc2aeb7ad647c6b1e"} Oct 30 00:25:06.813043 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:06.813016 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73ca7d2b4b6a79234763c75ee4ecf432e9830e814f44436cc2aeb7ad647c6b1e" Oct 30 00:25:06.813302 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:06.813283 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f52ea1-x2dh9" Oct 30 00:25:09.466066 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:09.465606 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-7f5fd784c9-pjrvt" Oct 30 00:25:11.760267 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:11.760229 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c34304-7ff4d749d8-fcpb6"] Oct 30 00:25:11.760881 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:11.760479 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-7ff4d749d8-fcpb6" podUID="51e1c913-e273-48ef-a8e0-4903c1b32b30" containerName="payload-processing" containerID="cri-o://2cefe897084883582a0f63911b97b8a2a3cd74cb1461d6809f0fd8ebea848bb6" gracePeriod=30 Oct 30 00:25:11.791750 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:11.791711 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-5dc97f76d7-9b4ng"] Oct 30 00:25:11.792299 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:11.792214 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-5dc97f76d7-9b4ng" podUID="bff05da7-3980-4634-be2f-6c3e1b160ec3" containerName="payload-pre-processing" containerID="cri-o://40ba89477e646f467218b54a010d1df96dc3c520dc214c48e0b415f1580f3eeb" gracePeriod=30 Oct 30 00:25:11.994351 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:11.994053 5001 generic.go:356] "Generic (PLEG): container finished" podID="bff05da7-3980-4634-be2f-6c3e1b160ec3" containerID="40ba89477e646f467218b54a010d1df96dc3c520dc214c48e0b415f1580f3eeb" exitCode=0 Oct 30 00:25:11.994351 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:11.994280 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-5dc97f76d7-9b4ng" event={"ID":"bff05da7-3980-4634-be2f-6c3e1b160ec3","Type":"ContainerDied","Data":"40ba89477e646f467218b54a010d1df96dc3c520dc214c48e0b415f1580f3eeb"} Oct 30 00:25:11.996701 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:11.996454 5001 generic.go:356] "Generic (PLEG): container finished" podID="51e1c913-e273-48ef-a8e0-4903c1b32b30" containerID="2cefe897084883582a0f63911b97b8a2a3cd74cb1461d6809f0fd8ebea848bb6" exitCode=0 Oct 30 00:25:11.996701 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:11.996620 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-7ff4d749d8-fcpb6" event={"ID":"51e1c913-e273-48ef-a8e0-4903c1b32b30","Type":"ContainerDied","Data":"2cefe897084883582a0f63911b97b8a2a3cd74cb1461d6809f0fd8ebea848bb6"} Oct 30 00:25:12.660147 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:12.660109 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e94c9f1-5pg6m"] Oct 30 00:25:12.704899 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:12.704812 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e94c9f1-5pg6m"] Oct 30 00:25:12.772768 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:12.772709 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c34304-glt92"] Oct 30 00:25:12.846123 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:12.846089 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c34304-glt92"] Oct 30 00:25:12.893627 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:12.893601 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-7ff4d749d8-fcpb6" Oct 30 00:25:12.927267 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:12.927173 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="442d9e02-2a7b-4573-918d-9e0c39bbed23" path="/var/lib/kubelet/pods/442d9e02-2a7b-4573-918d-9e0c39bbed23/volumes" Oct 30 00:25:12.927825 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:12.927802 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65ff7382-9b32-4c01-9d13-5df1841b0db6" path="/var/lib/kubelet/pods/65ff7382-9b32-4c01-9d13-5df1841b0db6/volumes" Oct 30 00:25:12.958651 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:12.958620 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51e1c913-e273-48ef-a8e0-4903c1b32b30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e1c913-e273-48ef-a8e0-4903c1b32b30-plugins-config-volume\") pod \"51e1c913-e273-48ef-a8e0-4903c1b32b30\" (UID: \"51e1c913-e273-48ef-a8e0-4903c1b32b30\") " Oct 30 00:25:12.958803 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:12.958728 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51e1c913-e273-48ef-a8e0-4903c1b32b30-kube-api-access-p5zwr\" (UniqueName: \"kubernetes.io/projected/51e1c913-e273-48ef-a8e0-4903c1b32b30-kube-api-access-p5zwr\") pod \"51e1c913-e273-48ef-a8e0-4903c1b32b30\" (UID: \"51e1c913-e273-48ef-a8e0-4903c1b32b30\") " Oct 30 00:25:12.959122 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:12.959093 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51e1c913-e273-48ef-a8e0-4903c1b32b30-plugins-config-volume" pod "51e1c913-e273-48ef-a8e0-4903c1b32b30" (UID: "51e1c913-e273-48ef-a8e0-4903c1b32b30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:25:12.959468 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:12.959446 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e1c913-e273-48ef-a8e0-4903c1b32b30-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:12.962434 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:12.962395 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e1c913-e273-48ef-a8e0-4903c1b32b30-kube-api-access-p5zwr" pod "51e1c913-e273-48ef-a8e0-4903c1b32b30" (UID: "51e1c913-e273-48ef-a8e0-4903c1b32b30"). InnerVolumeSpecName "kube-api-access-p5zwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:13.004313 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:13.004286 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-5dc97f76d7-9b4ng" Oct 30 00:25:13.013848 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:13.013804 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-5dc97f76d7-9b4ng" event={"ID":"bff05da7-3980-4634-be2f-6c3e1b160ec3","Type":"ContainerDied","Data":"b498075a08be50a5367b1399a7e565a421e25656a54fbd5d472a3922ffcf5822"} Oct 30 00:25:13.014121 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:13.014101 5001 scope.go:122] "RemoveContainer" containerID="40ba89477e646f467218b54a010d1df96dc3c520dc214c48e0b415f1580f3eeb" Oct 30 00:25:13.014456 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:13.014437 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-5dc97f76d7-9b4ng" Oct 30 00:25:13.023275 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:13.023238 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-7ff4d749d8-fcpb6" event={"ID":"51e1c913-e273-48ef-a8e0-4903c1b32b30","Type":"ContainerDied","Data":"72cc003670b42db9d29cd61cc044de430e2d842b5c4785f5cb2be5df27220de2"} Oct 30 00:25:13.023392 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:13.023354 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c34304-7ff4d749d8-fcpb6" Oct 30 00:25:13.035438 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:13.035423 5001 scope.go:122] "RemoveContainer" containerID="2cefe897084883582a0f63911b97b8a2a3cd74cb1461d6809f0fd8ebea848bb6" Oct 30 00:25:13.061779 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:13.061754 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5zwr\" (UniqueName: \"kubernetes.io/projected/51e1c913-e273-48ef-a8e0-4903c1b32b30-kube-api-access-p5zwr\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:13.163332 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:13.163303 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bff05da7-3980-4634-be2f-6c3e1b160ec3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bff05da7-3980-4634-be2f-6c3e1b160ec3-plugins-config-volume\") pod \"bff05da7-3980-4634-be2f-6c3e1b160ec3\" (UID: \"bff05da7-3980-4634-be2f-6c3e1b160ec3\") " Oct 30 00:25:13.163512 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:13.163464 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bff05da7-3980-4634-be2f-6c3e1b160ec3-kube-api-access-nqpj5\" (UniqueName: \"kubernetes.io/projected/bff05da7-3980-4634-be2f-6c3e1b160ec3-kube-api-access-nqpj5\") pod \"bff05da7-3980-4634-be2f-6c3e1b160ec3\" (UID: \"bff05da7-3980-4634-be2f-6c3e1b160ec3\") " Oct 30 00:25:13.163675 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:13.163651 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bff05da7-3980-4634-be2f-6c3e1b160ec3-plugins-config-volume" pod "bff05da7-3980-4634-be2f-6c3e1b160ec3" (UID: "bff05da7-3980-4634-be2f-6c3e1b160ec3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:25:13.164385 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:13.164362 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bff05da7-3980-4634-be2f-6c3e1b160ec3-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:13.165893 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:13.165862 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bff05da7-3980-4634-be2f-6c3e1b160ec3-kube-api-access-nqpj5" pod "bff05da7-3980-4634-be2f-6c3e1b160ec3" (UID: "bff05da7-3980-4634-be2f-6c3e1b160ec3"). InnerVolumeSpecName "kube-api-access-nqpj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:13.173553 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:13.173527 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c34304-7ff4d749d8-fcpb6"] Oct 30 00:25:13.182587 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:13.182560 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c34304-7ff4d749d8-fcpb6"] Oct 30 00:25:13.266776 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:13.266738 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqpj5\" (UniqueName: \"kubernetes.io/projected/bff05da7-3980-4634-be2f-6c3e1b160ec3-kube-api-access-nqpj5\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:13.362888 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:13.362853 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-5dc97f76d7-9b4ng"] Oct 30 00:25:13.368504 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:13.368481 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-5dc97f76d7-9b4ng"] Oct 30 00:25:13.932064 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:13.930928 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c34304-5dc97f76d7-9b4ng" podUID="bff05da7-3980-4634-be2f-6c3e1b160ec3" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.159:9004: i/o timeout" Oct 30 00:25:14.213622 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:14.210676 5001 generic.go:356] "Generic (PLEG): container finished" podID="03f8426e-b14c-43f9-9598-04b5b7b309b5" containerID="13b59302f73b01109636fec53bf8d139d4c2af5430bf221ebba254a665d0294c" exitCode=0 Oct 30 00:25:14.213622 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:14.210882 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" event={"ID":"03f8426e-b14c-43f9-9598-04b5b7b309b5","Type":"ContainerDied","Data":"13b59302f73b01109636fec53bf8d139d4c2af5430bf221ebba254a665d0294c"} Oct 30 00:25:14.213622 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:14.211558 5001 scope.go:122] "RemoveContainer" containerID="13b59302f73b01109636fec53bf8d139d4c2af5430bf221ebba254a665d0294c" Oct 30 00:25:14.216402 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:14.215638 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:25:14.216402 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:14.215733 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:25:14.216402 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:14.215784 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:25:14.349998 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:14.349957 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-hw97x"] Oct 30 00:25:14.359762 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:14.359702 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0847e8c0-hw97x"] Oct 30 00:25:15.042936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:15.036125 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51e1c913-e273-48ef-a8e0-4903c1b32b30" path="/var/lib/kubelet/pods/51e1c913-e273-48ef-a8e0-4903c1b32b30/volumes" Oct 30 00:25:15.042936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:15.037313 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a83e6d25-723d-48ed-9679-565e3953acce" path="/var/lib/kubelet/pods/a83e6d25-723d-48ed-9679-565e3953acce/volumes" Oct 30 00:25:15.042936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:15.037939 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bff05da7-3980-4634-be2f-6c3e1b160ec3" path="/var/lib/kubelet/pods/bff05da7-3980-4634-be2f-6c3e1b160ec3/volumes" Oct 30 00:25:15.456032 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:15.455145 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" event={"ID":"03f8426e-b14c-43f9-9598-04b5b7b309b5","Type":"ContainerStarted","Data":"cf1f6bee7420b91495207ae3fd2dabf726fc1481aaac039fc291bcf9ef71b8a4"} Oct 30 00:25:16.165604 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:16.165572 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:25:16.166446 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:16.166413 5001 patch_prober.go:29] interesting pod/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 00:25:16.166585 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:16.166466 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" podUID="03f8426e-b14c-43f9-9598-04b5b7b309b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 00:25:16.768561 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:16.768524 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:25:17.165297 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:17.165210 5001 patch_prober.go:29] interesting pod/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 00:25:17.170835 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:17.170795 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" podUID="03f8426e-b14c-43f9-9598-04b5b7b309b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 00:25:17.548299 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:25:17.548250 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:25:18.166108 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:18.166076 5001 patch_prober.go:29] interesting pod/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 00:25:18.166294 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:18.166130 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" podUID="03f8426e-b14c-43f9-9598-04b5b7b309b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 00:25:19.194746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:19.194285 5001 patch_prober.go:29] interesting pod/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 00:25:19.194746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:19.194351 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" podUID="03f8426e-b14c-43f9-9598-04b5b7b309b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 00:25:19.943513 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:19.943472 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9"] Oct 30 00:25:20.007938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:19.966231 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" podUID="4bfbc166-3dbf-4aae-96f5-515f59917846" containerName="istio-proxy" containerID="cri-o://c4f68de95ea8a0fe7c2ab9ab7df309f72eb2d21accc0ed06ee4d1a7700da57e9" gracePeriod=30 Oct 30 00:25:20.101007 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:20.100969 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds"] Oct 30 00:25:20.171135 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:20.169886 5001 patch_prober.go:29] interesting pod/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 00:25:20.171135 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:20.169980 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" podUID="03f8426e-b14c-43f9-9598-04b5b7b309b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 00:25:20.251011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:20.245830 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:25:20.420934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:20.416993 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:25:20.690938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:20.685399 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96027e-gn8cz"] Oct 30 00:25:20.690938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:20.687394 5001 cpu_manager.go:414] "containerMap: removing container" podUID="bff05da7-3980-4634-be2f-6c3e1b160ec3" containerName="payload-pre-processing" Oct 30 00:25:20.690938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:20.687420 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bff05da7-3980-4634-be2f-6c3e1b160ec3" containerName="payload-pre-processing" Oct 30 00:25:20.690938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:20.687450 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1da01660-8624-4707-bd34-edba0f103f2f" containerName="revoke-keys" Oct 30 00:25:20.690938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:20.687458 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1da01660-8624-4707-bd34-edba0f103f2f" containerName="revoke-keys" Oct 30 00:25:20.690938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:20.687531 5001 cpu_manager.go:414] "containerMap: removing container" podUID="68d5a825-45b1-45ec-9d34-1b141f6e6daa" containerName="revoke-keys" Oct 30 00:25:20.690938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:20.687539 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68d5a825-45b1-45ec-9d34-1b141f6e6daa" containerName="revoke-keys" Oct 30 00:25:20.690938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:20.687594 5001 cpu_manager.go:414] "containerMap: removing container" podUID="442d9e02-2a7b-4573-918d-9e0c39bbed23" containerName="revoke-keys" Oct 30 00:25:20.690938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:20.687602 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="442d9e02-2a7b-4573-918d-9e0c39bbed23" containerName="revoke-keys" Oct 30 00:25:20.690938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:20.687624 5001 cpu_manager.go:414] "containerMap: removing container" podUID="51e1c913-e273-48ef-a8e0-4903c1b32b30" containerName="payload-processing" Oct 30 00:25:20.690938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:20.687631 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51e1c913-e273-48ef-a8e0-4903c1b32b30" containerName="payload-processing" Oct 30 00:25:20.839011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:20.838239 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="bff05da7-3980-4634-be2f-6c3e1b160ec3" containerName="payload-pre-processing" Oct 30 00:25:20.839011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:20.838280 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1da01660-8624-4707-bd34-edba0f103f2f" containerName="revoke-keys" Oct 30 00:25:20.839011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:20.838328 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="68d5a825-45b1-45ec-9d34-1b141f6e6daa" containerName="revoke-keys" Oct 30 00:25:20.839011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:20.838358 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="442d9e02-2a7b-4573-918d-9e0c39bbed23" containerName="revoke-keys" Oct 30 00:25:20.839011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:20.838375 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="51e1c913-e273-48ef-a8e0-4903c1b32b30" containerName="payload-processing" Oct 30 00:25:20.890958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:20.886001 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:25:20.890958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:20.886134 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" podUID="4bfbc166-3dbf-4aae-96f5-515f59917846" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:25:20.923937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:20.920215 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" podUID="b3e66631-9562-4bb5-b005-1fc179a8582d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.135:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:25:20.950587 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:20.950500 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96027e-gn8cz" Oct 30 00:25:20.976935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:20.971979 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:25:20.984516 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:20.984265 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:25:20.984516 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:20.984384 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:25:20.984516 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:20.984450 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:25:20.984788 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:20.984551 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:25:20.999014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:20.998988 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96027e-gn8cz"] Oct 30 00:25:21.049001 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:21.048972 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-96027e-gn8cz\" (UID: \"c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96027e-gn8cz" Oct 30 00:25:21.049162 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:21.049026 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-96027e-gn8cz\" (UID: \"c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96027e-gn8cz" Oct 30 00:25:21.049162 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:21.049111 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhpph\" (UniqueName: \"kubernetes.io/projected/c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8-kube-api-access-nhpph\") pod \"maas-api-revoke-keys-e2e-ipp-b-96027e-gn8cz\" (UID: \"c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96027e-gn8cz" Oct 30 00:25:21.150614 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:21.150579 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-96027e-gn8cz\" (UID: \"c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96027e-gn8cz" Oct 30 00:25:21.150807 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:21.150639 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-96027e-gn8cz\" (UID: \"c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96027e-gn8cz" Oct 30 00:25:21.150807 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:21.150703 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhpph\" (UniqueName: \"kubernetes.io/projected/c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8-kube-api-access-nhpph\") pod \"maas-api-revoke-keys-e2e-ipp-b-96027e-gn8cz\" (UID: \"c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96027e-gn8cz" Oct 30 00:25:21.151445 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:21.151420 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-96027e-gn8cz\" (UID: \"c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96027e-gn8cz" Oct 30 00:25:21.153046 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:21.153020 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-96027e-gn8cz\" (UID: \"c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96027e-gn8cz" Oct 30 00:25:21.165113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:21.165091 5001 patch_prober.go:29] interesting pod/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 00:25:21.165210 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:21.165149 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" podUID="03f8426e-b14c-43f9-9598-04b5b7b309b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 00:25:21.171463 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:21.171444 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhpph\" (UniqueName: \"kubernetes.io/projected/c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8-kube-api-access-nhpph\") pod \"maas-api-revoke-keys-e2e-ipp-b-96027e-gn8cz\" (UID: \"c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96027e-gn8cz" Oct 30 00:25:21.293980 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:21.293949 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96027e-gn8cz" Oct 30 00:25:21.748365 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:21.747968 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96027e-gn8cz"] Oct 30 00:25:22.004836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:22.004754 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96027e-gn8cz" event={"ID":"c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8","Type":"ContainerStarted","Data":"17ecb5b776bf02b2818022c6264b6fee7f4f6408569e19312b583f21ac41efd3"} Oct 30 00:25:22.179967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:22.178243 5001 patch_prober.go:29] interesting pod/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 00:25:22.179967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:22.178311 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" podUID="03f8426e-b14c-43f9-9598-04b5b7b309b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 00:25:23.016705 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:23.016657 5001 generic.go:356] "Generic (PLEG): container finished" podID="c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8" containerID="40a9e950830a91bf6053e823c280f93823258336a2915ab28e56ee03dbf124d6" exitCode=0 Oct 30 00:25:23.017106 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:23.016697 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96027e-gn8cz" event={"ID":"c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8","Type":"ContainerDied","Data":"40a9e950830a91bf6053e823c280f93823258336a2915ab28e56ee03dbf124d6"} Oct 30 00:25:23.169802 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:23.169731 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:25:23.170895 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:23.170870 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:25:24.220712 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:24.220687 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96027e-gn8cz" Oct 30 00:25:24.331245 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:24.331165 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8-kube-api-access-nhpph\" (UniqueName: \"kubernetes.io/projected/c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8-kube-api-access-nhpph\") pod \"c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8\" (UID: \"c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8\") " Oct 30 00:25:24.331399 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:24.331305 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8-maas-api-xks-ca\") pod \"c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8\" (UID: \"c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8\") " Oct 30 00:25:24.331399 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:24.331376 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8-maas-api-openshift-ca\") pod \"c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8\" (UID: \"c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8\") " Oct 30 00:25:24.331750 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:24.331720 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8-maas-api-openshift-ca" pod "c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8" (UID: "c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:25:24.331943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:24.331903 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:24.333312 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:24.333290 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8-maas-api-xks-ca" pod "c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8" (UID: "c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:25:24.333465 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:24.333445 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8-kube-api-access-nhpph" pod "c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8" (UID: "c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8"). InnerVolumeSpecName "kube-api-access-nhpph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:24.433611 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:24.433576 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhpph\" (UniqueName: \"kubernetes.io/projected/c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8-kube-api-access-nhpph\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:24.433611 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:24.433611 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:24.781266 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:24.781225 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-7dcd79d5b6-ssgxn"] Oct 30 00:25:24.781494 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:24.781468 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-7dcd79d5b6-ssgxn" podUID="373a854c-4772-4632-b9fe-813747f39ca9" containerName="payload-processing" containerID="cri-o://81c802c8474f3d3cf0066d52bb9c8b9ececb180e37934f2920c9ed41fc00a5f3" gracePeriod=30 Oct 30 00:25:24.808352 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:24.808323 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-58dbc8bd4c-44p8c"] Oct 30 00:25:24.808586 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:24.808550 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-58dbc8bd4c-44p8c" podUID="a4253945-fb6f-438c-937e-b5061e33c84b" containerName="payload-pre-processing" containerID="cri-o://388d571c14353142344f986aef92bac36806e0e1aa7e04d320bd6d21412d9a1b" gracePeriod=30 Oct 30 00:25:25.353791 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:25.352314 5001 generic.go:356] "Generic (PLEG): container finished" podID="a4253945-fb6f-438c-937e-b5061e33c84b" containerID="388d571c14353142344f986aef92bac36806e0e1aa7e04d320bd6d21412d9a1b" exitCode=0 Oct 30 00:25:25.353791 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:25.352582 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-58dbc8bd4c-44p8c" event={"ID":"a4253945-fb6f-438c-937e-b5061e33c84b","Type":"ContainerDied","Data":"388d571c14353142344f986aef92bac36806e0e1aa7e04d320bd6d21412d9a1b"} Oct 30 00:25:25.360517 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:25.360046 5001 generic.go:356] "Generic (PLEG): container finished" podID="373a854c-4772-4632-b9fe-813747f39ca9" containerID="81c802c8474f3d3cf0066d52bb9c8b9ececb180e37934f2920c9ed41fc00a5f3" exitCode=0 Oct 30 00:25:25.360517 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:25.360405 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-7dcd79d5b6-ssgxn" event={"ID":"373a854c-4772-4632-b9fe-813747f39ca9","Type":"ContainerDied","Data":"81c802c8474f3d3cf0066d52bb9c8b9ececb180e37934f2920c9ed41fc00a5f3"} Oct 30 00:25:25.372641 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:25.372556 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96027e-gn8cz" event={"ID":"c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8","Type":"ContainerDied","Data":"17ecb5b776bf02b2818022c6264b6fee7f4f6408569e19312b583f21ac41efd3"} Oct 30 00:25:25.373012 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:25.372975 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17ecb5b776bf02b2818022c6264b6fee7f4f6408569e19312b583f21ac41efd3" Oct 30 00:25:25.373251 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:25.373228 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96027e-gn8cz" Oct 30 00:25:25.722139 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:25.721986 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f52ea1-x2dh9"] Oct 30 00:25:25.744958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:25.730844 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f52ea1-x2dh9"] Oct 30 00:25:25.815860 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:25.815440 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-7dcd79d5b6-ssgxn" Oct 30 00:25:25.850889 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:25.850863 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-58dbc8bd4c-44p8c" Oct 30 00:25:25.934983 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:25.931348 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4253945-fb6f-438c-937e-b5061e33c84b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4253945-fb6f-438c-937e-b5061e33c84b-plugins-config-volume\") pod \"a4253945-fb6f-438c-937e-b5061e33c84b\" (UID: \"a4253945-fb6f-438c-937e-b5061e33c84b\") " Oct 30 00:25:25.934983 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:25.931511 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/373a854c-4772-4632-b9fe-813747f39ca9-kube-api-access-c48wk\" (UniqueName: \"kubernetes.io/projected/373a854c-4772-4632-b9fe-813747f39ca9-kube-api-access-c48wk\") pod \"373a854c-4772-4632-b9fe-813747f39ca9\" (UID: \"373a854c-4772-4632-b9fe-813747f39ca9\") " Oct 30 00:25:25.934983 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:25.931563 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4253945-fb6f-438c-937e-b5061e33c84b-kube-api-access-w6nj8\" (UniqueName: \"kubernetes.io/projected/a4253945-fb6f-438c-937e-b5061e33c84b-kube-api-access-w6nj8\") pod \"a4253945-fb6f-438c-937e-b5061e33c84b\" (UID: \"a4253945-fb6f-438c-937e-b5061e33c84b\") " Oct 30 00:25:25.934983 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:25.931788 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/373a854c-4772-4632-b9fe-813747f39ca9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373a854c-4772-4632-b9fe-813747f39ca9-plugins-config-volume\") pod \"373a854c-4772-4632-b9fe-813747f39ca9\" (UID: \"373a854c-4772-4632-b9fe-813747f39ca9\") " Oct 30 00:25:25.944783 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:25.944725 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4253945-fb6f-438c-937e-b5061e33c84b-plugins-config-volume" pod "a4253945-fb6f-438c-937e-b5061e33c84b" (UID: "a4253945-fb6f-438c-937e-b5061e33c84b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:25:25.950703 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:25.950644 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/373a854c-4772-4632-b9fe-813747f39ca9-plugins-config-volume" pod "373a854c-4772-4632-b9fe-813747f39ca9" (UID: "373a854c-4772-4632-b9fe-813747f39ca9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:25:25.952256 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:25.952226 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373a854c-4772-4632-b9fe-813747f39ca9-kube-api-access-c48wk" pod "373a854c-4772-4632-b9fe-813747f39ca9" (UID: "373a854c-4772-4632-b9fe-813747f39ca9"). InnerVolumeSpecName "kube-api-access-c48wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:25.954687 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:25.954641 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4253945-fb6f-438c-937e-b5061e33c84b-kube-api-access-w6nj8" pod "a4253945-fb6f-438c-937e-b5061e33c84b" (UID: "a4253945-fb6f-438c-937e-b5061e33c84b"). InnerVolumeSpecName "kube-api-access-w6nj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:25.955424 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:25.955404 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4253945-fb6f-438c-937e-b5061e33c84b-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:25.955583 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:25.955568 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c48wk\" (UniqueName: \"kubernetes.io/projected/373a854c-4772-4632-b9fe-813747f39ca9-kube-api-access-c48wk\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:25.955713 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:25.955684 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6nj8\" (UniqueName: \"kubernetes.io/projected/a4253945-fb6f-438c-937e-b5061e33c84b-kube-api-access-w6nj8\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:25.955818 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:25.955805 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373a854c-4772-4632-b9fe-813747f39ca9-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:26.087652 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.087625 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:25:26.158983 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.158945 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-workload-certs\") pod \"4bfbc166-3dbf-4aae-96f5-515f59917846\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " Oct 30 00:25:26.159176 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.159006 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bfbc166-3dbf-4aae-96f5-515f59917846-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bfbc166-3dbf-4aae-96f5-515f59917846-istiod-ca-cert\") pod \"4bfbc166-3dbf-4aae-96f5-515f59917846\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " Oct 30 00:25:26.159176 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.159125 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bfbc166-3dbf-4aae-96f5-515f59917846-istio-token\" (UniqueName: \"kubernetes.io/projected/4bfbc166-3dbf-4aae-96f5-515f59917846-istio-token\") pod \"4bfbc166-3dbf-4aae-96f5-515f59917846\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " Oct 30 00:25:26.159321 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.159257 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-workload-certs" pod "4bfbc166-3dbf-4aae-96f5-515f59917846" (UID: "4bfbc166-3dbf-4aae-96f5-515f59917846"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:26.159321 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.159304 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4bfbc166-3dbf-4aae-96f5-515f59917846-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bfbc166-3dbf-4aae-96f5-515f59917846-istio-podinfo\") pod \"4bfbc166-3dbf-4aae-96f5-515f59917846\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " Oct 30 00:25:26.159438 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.159352 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bfbc166-3dbf-4aae-96f5-515f59917846-kube-api-access-kh2xd\" (UniqueName: \"kubernetes.io/projected/4bfbc166-3dbf-4aae-96f5-515f59917846-kube-api-access-kh2xd\") pod \"4bfbc166-3dbf-4aae-96f5-515f59917846\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " Oct 30 00:25:26.159438 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.159421 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-credential-socket\") pod \"4bfbc166-3dbf-4aae-96f5-515f59917846\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " Oct 30 00:25:26.159538 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.159499 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-istio-envoy\") pod \"4bfbc166-3dbf-4aae-96f5-515f59917846\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " Oct 30 00:25:26.159538 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.159527 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-workload-socket\") pod \"4bfbc166-3dbf-4aae-96f5-515f59917846\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " Oct 30 00:25:26.159633 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.159555 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-istio-data\") pod \"4bfbc166-3dbf-4aae-96f5-515f59917846\" (UID: \"4bfbc166-3dbf-4aae-96f5-515f59917846\") " Oct 30 00:25:26.159929 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.159416 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bfbc166-3dbf-4aae-96f5-515f59917846-istiod-ca-cert" pod "4bfbc166-3dbf-4aae-96f5-515f59917846" (UID: "4bfbc166-3dbf-4aae-96f5-515f59917846"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:25:26.160205 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.159655 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-credential-socket" pod "4bfbc166-3dbf-4aae-96f5-515f59917846" (UID: "4bfbc166-3dbf-4aae-96f5-515f59917846"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:26.160205 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.160162 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-istio-data" pod "4bfbc166-3dbf-4aae-96f5-515f59917846" (UID: "4bfbc166-3dbf-4aae-96f5-515f59917846"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:26.160369 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.160345 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-workload-socket" pod "4bfbc166-3dbf-4aae-96f5-515f59917846" (UID: "4bfbc166-3dbf-4aae-96f5-515f59917846"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:26.160868 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.160843 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:26.161021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.160873 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:26.161021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.160889 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:26.161021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.160902 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:26.161021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.160932 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bfbc166-3dbf-4aae-96f5-515f59917846-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:26.161685 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.161663 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bfbc166-3dbf-4aae-96f5-515f59917846-kube-api-access-kh2xd" pod "4bfbc166-3dbf-4aae-96f5-515f59917846" (UID: "4bfbc166-3dbf-4aae-96f5-515f59917846"). InnerVolumeSpecName "kube-api-access-kh2xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:26.161823 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.161801 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bfbc166-3dbf-4aae-96f5-515f59917846-istio-podinfo" pod "4bfbc166-3dbf-4aae-96f5-515f59917846" (UID: "4bfbc166-3dbf-4aae-96f5-515f59917846"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:25:26.161875 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.161839 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bfbc166-3dbf-4aae-96f5-515f59917846-istio-token" pod "4bfbc166-3dbf-4aae-96f5-515f59917846" (UID: "4bfbc166-3dbf-4aae-96f5-515f59917846"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:26.162399 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.162377 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-istio-envoy" pod "4bfbc166-3dbf-4aae-96f5-515f59917846" (UID: "4bfbc166-3dbf-4aae-96f5-515f59917846"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:26.262656 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.262621 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bfbc166-3dbf-4aae-96f5-515f59917846-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:26.262656 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.262651 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bfbc166-3dbf-4aae-96f5-515f59917846-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:26.262656 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.262662 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh2xd\" (UniqueName: \"kubernetes.io/projected/4bfbc166-3dbf-4aae-96f5-515f59917846-kube-api-access-kh2xd\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:26.262885 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.262673 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bfbc166-3dbf-4aae-96f5-515f59917846-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:26.384308 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.384272 5001 generic.go:356] "Generic (PLEG): container finished" podID="4bfbc166-3dbf-4aae-96f5-515f59917846" containerID="c4f68de95ea8a0fe7c2ab9ab7df309f72eb2d21accc0ed06ee4d1a7700da57e9" exitCode=0 Oct 30 00:25:26.384725 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.384347 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" Oct 30 00:25:26.384725 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.384357 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" event={"ID":"4bfbc166-3dbf-4aae-96f5-515f59917846","Type":"ContainerDied","Data":"c4f68de95ea8a0fe7c2ab9ab7df309f72eb2d21accc0ed06ee4d1a7700da57e9"} Oct 30 00:25:26.384725 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.384434 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9" event={"ID":"4bfbc166-3dbf-4aae-96f5-515f59917846","Type":"ContainerDied","Data":"50149fb4715117fa618e2d60cce683f67959d04e7b3f2b68092af06fe4e38a43"} Oct 30 00:25:26.384725 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.384457 5001 scope.go:122] "RemoveContainer" containerID="c4f68de95ea8a0fe7c2ab9ab7df309f72eb2d21accc0ed06ee4d1a7700da57e9" Oct 30 00:25:26.386166 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.386145 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-58dbc8bd4c-44p8c" Oct 30 00:25:26.386297 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.386170 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-58dbc8bd4c-44p8c" event={"ID":"a4253945-fb6f-438c-937e-b5061e33c84b","Type":"ContainerDied","Data":"585795ab829901416c950fb44c1bb364d5adacfe3654f80e126e90faa6c456a9"} Oct 30 00:25:26.388350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.388320 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-7dcd79d5b6-ssgxn" event={"ID":"373a854c-4772-4632-b9fe-813747f39ca9","Type":"ContainerDied","Data":"49c8be211b13cc96d59c7dd3bc46e7ece2e6a3e5026810a9c944ea1f0509f635"} Oct 30 00:25:26.388350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.388345 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-7dcd79d5b6-ssgxn" Oct 30 00:25:26.401602 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.401588 5001 scope.go:122] "RemoveContainer" containerID="af07da8cc8da63de4de9af460ea429fc7f10f344885d5ee21a8d9d1a291b8d3d" Oct 30 00:25:26.412669 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.412650 5001 scope.go:122] "RemoveContainer" containerID="c4f68de95ea8a0fe7c2ab9ab7df309f72eb2d21accc0ed06ee4d1a7700da57e9" Oct 30 00:25:26.412970 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:25:26.412935 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4f68de95ea8a0fe7c2ab9ab7df309f72eb2d21accc0ed06ee4d1a7700da57e9\": container with ID starting with c4f68de95ea8a0fe7c2ab9ab7df309f72eb2d21accc0ed06ee4d1a7700da57e9 not found: ID does not exist" containerID="c4f68de95ea8a0fe7c2ab9ab7df309f72eb2d21accc0ed06ee4d1a7700da57e9" Oct 30 00:25:26.413041 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.412982 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4f68de95ea8a0fe7c2ab9ab7df309f72eb2d21accc0ed06ee4d1a7700da57e9"} err="failed to get container status \"c4f68de95ea8a0fe7c2ab9ab7df309f72eb2d21accc0ed06ee4d1a7700da57e9\": rpc error: code = NotFound desc = could not find container \"c4f68de95ea8a0fe7c2ab9ab7df309f72eb2d21accc0ed06ee4d1a7700da57e9\": container with ID starting with c4f68de95ea8a0fe7c2ab9ab7df309f72eb2d21accc0ed06ee4d1a7700da57e9 not found: ID does not exist" Oct 30 00:25:26.413041 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.413005 5001 scope.go:122] "RemoveContainer" containerID="af07da8cc8da63de4de9af460ea429fc7f10f344885d5ee21a8d9d1a291b8d3d" Oct 30 00:25:26.413255 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:25:26.413236 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af07da8cc8da63de4de9af460ea429fc7f10f344885d5ee21a8d9d1a291b8d3d\": container with ID starting with af07da8cc8da63de4de9af460ea429fc7f10f344885d5ee21a8d9d1a291b8d3d not found: ID does not exist" containerID="af07da8cc8da63de4de9af460ea429fc7f10f344885d5ee21a8d9d1a291b8d3d" Oct 30 00:25:26.413302 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.413261 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af07da8cc8da63de4de9af460ea429fc7f10f344885d5ee21a8d9d1a291b8d3d"} err="failed to get container status \"af07da8cc8da63de4de9af460ea429fc7f10f344885d5ee21a8d9d1a291b8d3d\": rpc error: code = NotFound desc = could not find container \"af07da8cc8da63de4de9af460ea429fc7f10f344885d5ee21a8d9d1a291b8d3d\": container with ID starting with af07da8cc8da63de4de9af460ea429fc7f10f344885d5ee21a8d9d1a291b8d3d not found: ID does not exist" Oct 30 00:25:26.413302 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.413279 5001 scope.go:122] "RemoveContainer" containerID="388d571c14353142344f986aef92bac36806e0e1aa7e04d320bd6d21412d9a1b" Oct 30 00:25:26.424949 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.424893 5001 scope.go:122] "RemoveContainer" containerID="81c802c8474f3d3cf0066d52bb9c8b9ececb180e37934f2920c9ed41fc00a5f3" Oct 30 00:25:26.456925 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.456891 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-58dbc8bd4c-44p8c"] Oct 30 00:25:26.462361 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.462311 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f52ea1-58dbc8bd4c-44p8c"] Oct 30 00:25:26.486261 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.486233 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-7dcd79d5b6-ssgxn"] Oct 30 00:25:26.491354 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.491335 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f52ea1-7dcd79d5b6-ssgxn"] Oct 30 00:25:26.518342 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.518317 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9"] Oct 30 00:25:26.531612 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.531590 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c34304-openshift-default-58676bd45-vzpp9"] Oct 30 00:25:26.921117 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.921082 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="373a854c-4772-4632-b9fe-813747f39ca9" path="/var/lib/kubelet/pods/373a854c-4772-4632-b9fe-813747f39ca9/volumes" Oct 30 00:25:26.921735 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.921712 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bfbc166-3dbf-4aae-96f5-515f59917846" path="/var/lib/kubelet/pods/4bfbc166-3dbf-4aae-96f5-515f59917846/volumes" Oct 30 00:25:26.922295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.922274 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68d5a825-45b1-45ec-9d34-1b141f6e6daa" path="/var/lib/kubelet/pods/68d5a825-45b1-45ec-9d34-1b141f6e6daa/volumes" Oct 30 00:25:26.923079 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:26.923059 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4253945-fb6f-438c-937e-b5061e33c84b" path="/var/lib/kubelet/pods/a4253945-fb6f-438c-937e-b5061e33c84b/volumes" Oct 30 00:25:29.031534 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:29.031501 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b76cbf84-8p4mk"] Oct 30 00:25:29.041138 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:29.041107 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2dd5d03-zgjzc"] Oct 30 00:25:29.046685 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:29.046659 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b76cbf84-8p4mk"] Oct 30 00:25:29.051780 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:29.051749 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2dd5d03-zgjzc"] Oct 30 00:25:30.920553 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:30.920521 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="304c6e56-6ae0-4f85-b8f7-cad3069bcf50" path="/var/lib/kubelet/pods/304c6e56-6ae0-4f85-b8f7-cad3069bcf50/volumes" Oct 30 00:25:30.921069 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:30.921053 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f69d870-15fa-4208-abca-2bfa331060a1" path="/var/lib/kubelet/pods/6f69d870-15fa-4208-abca-2bfa331060a1/volumes" Oct 30 00:25:32.760866 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:32.760828 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4"] Oct 30 00:25:32.762585 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:32.762549 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" podUID="ed3d03e4-5550-40bc-8c00-48c3c7927fa1" containerName="istio-proxy" containerID="cri-o://236dd29a33f1bc6f9edc1a3037b646e0ee769237581b5c6333a209b2877f3f3f" gracePeriod=30 Oct 30 00:25:35.992019 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:35.991978 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" podUID="b3e66631-9562-4bb5-b005-1fc179a8582d" containerName="main" containerID="cri-o://ec17342524a50c2c5323dda527cdfddeb0d6c6b672677176bf4892faa6da465d" gracePeriod=45 Oct 30 00:25:36.030608 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:36.030580 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53c6eb88-thnxg"] Oct 30 00:25:36.039144 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:36.039119 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53c6eb88-thnxg"] Oct 30 00:25:36.289163 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:36.289137 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" Oct 30 00:25:36.450887 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:36.449615 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3e66631-9562-4bb5-b005-1fc179a8582d-kube-api-access-mvbdl\" (UniqueName: \"kubernetes.io/projected/b3e66631-9562-4bb5-b005-1fc179a8582d-kube-api-access-mvbdl\") pod \"b3e66631-9562-4bb5-b005-1fc179a8582d\" (UID: \"b3e66631-9562-4bb5-b005-1fc179a8582d\") " Oct 30 00:25:36.450887 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:36.449788 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3e66631-9562-4bb5-b005-1fc179a8582d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3e66631-9562-4bb5-b005-1fc179a8582d-model-cache\") pod \"b3e66631-9562-4bb5-b005-1fc179a8582d\" (UID: \"b3e66631-9562-4bb5-b005-1fc179a8582d\") " Oct 30 00:25:36.450887 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:36.449834 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3e66631-9562-4bb5-b005-1fc179a8582d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3e66631-9562-4bb5-b005-1fc179a8582d-dshm\") pod \"b3e66631-9562-4bb5-b005-1fc179a8582d\" (UID: \"b3e66631-9562-4bb5-b005-1fc179a8582d\") " Oct 30 00:25:36.450887 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:36.449981 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3e66631-9562-4bb5-b005-1fc179a8582d-tls-certs\" (UniqueName: \"kubernetes.io/secret/b3e66631-9562-4bb5-b005-1fc179a8582d-tls-certs\") pod \"b3e66631-9562-4bb5-b005-1fc179a8582d\" (UID: \"b3e66631-9562-4bb5-b005-1fc179a8582d\") " Oct 30 00:25:36.450887 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:36.450050 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3e66631-9562-4bb5-b005-1fc179a8582d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3e66631-9562-4bb5-b005-1fc179a8582d-tmp-dir\") pod \"b3e66631-9562-4bb5-b005-1fc179a8582d\" (UID: \"b3e66631-9562-4bb5-b005-1fc179a8582d\") " Oct 30 00:25:36.450887 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:36.450079 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3e66631-9562-4bb5-b005-1fc179a8582d-home\" (UniqueName: \"kubernetes.io/empty-dir/b3e66631-9562-4bb5-b005-1fc179a8582d-home\") pod \"b3e66631-9562-4bb5-b005-1fc179a8582d\" (UID: \"b3e66631-9562-4bb5-b005-1fc179a8582d\") " Oct 30 00:25:36.451554 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:36.451518 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3e66631-9562-4bb5-b005-1fc179a8582d-tmp-dir" pod "b3e66631-9562-4bb5-b005-1fc179a8582d" (UID: "b3e66631-9562-4bb5-b005-1fc179a8582d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:36.452587 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:36.452552 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3e66631-9562-4bb5-b005-1fc179a8582d-home" pod "b3e66631-9562-4bb5-b005-1fc179a8582d" (UID: "b3e66631-9562-4bb5-b005-1fc179a8582d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:36.452587 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:36.452568 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3e66631-9562-4bb5-b005-1fc179a8582d-model-cache" pod "b3e66631-9562-4bb5-b005-1fc179a8582d" (UID: "b3e66631-9562-4bb5-b005-1fc179a8582d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:36.454392 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:36.454359 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3e66631-9562-4bb5-b005-1fc179a8582d-tls-certs" pod "b3e66631-9562-4bb5-b005-1fc179a8582d" (UID: "b3e66631-9562-4bb5-b005-1fc179a8582d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:25:36.454499 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:36.454401 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3e66631-9562-4bb5-b005-1fc179a8582d-kube-api-access-mvbdl" pod "b3e66631-9562-4bb5-b005-1fc179a8582d" (UID: "b3e66631-9562-4bb5-b005-1fc179a8582d"). InnerVolumeSpecName "kube-api-access-mvbdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:36.454746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:36.454731 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3e66631-9562-4bb5-b005-1fc179a8582d-dshm" pod "b3e66631-9562-4bb5-b005-1fc179a8582d" (UID: "b3e66631-9562-4bb5-b005-1fc179a8582d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:36.552780 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:36.552745 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvbdl\" (UniqueName: \"kubernetes.io/projected/b3e66631-9562-4bb5-b005-1fc179a8582d-kube-api-access-mvbdl\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:36.552780 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:36.552775 5001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3e66631-9562-4bb5-b005-1fc179a8582d-model-cache\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:36.553011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:36.552799 5001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3e66631-9562-4bb5-b005-1fc179a8582d-dshm\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:36.553011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:36.552810 5001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3e66631-9562-4bb5-b005-1fc179a8582d-tls-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:36.553011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:36.552821 5001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3e66631-9562-4bb5-b005-1fc179a8582d-tmp-dir\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:36.553011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:36.552830 5001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3e66631-9562-4bb5-b005-1fc179a8582d-home\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:36.697923 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:36.697858 5001 generic.go:356] "Generic (PLEG): container finished" podID="b3e66631-9562-4bb5-b005-1fc179a8582d" containerID="ec17342524a50c2c5323dda527cdfddeb0d6c6b672677176bf4892faa6da465d" exitCode=0 Oct 30 00:25:36.698275 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:36.698080 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" Oct 30 00:25:36.699222 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:36.698099 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" event={"ID":"b3e66631-9562-4bb5-b005-1fc179a8582d","Type":"ContainerDied","Data":"ec17342524a50c2c5323dda527cdfddeb0d6c6b672677176bf4892faa6da465d"} Oct 30 00:25:36.699222 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:36.698512 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds" event={"ID":"b3e66631-9562-4bb5-b005-1fc179a8582d","Type":"ContainerDied","Data":"cdd4684c7c72f364c6a5b42bfc1fb246799f7be8829c919e35400234cf471016"} Oct 30 00:25:36.699222 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:36.698559 5001 scope.go:122] "RemoveContainer" containerID="ec17342524a50c2c5323dda527cdfddeb0d6c6b672677176bf4892faa6da465d" Oct 30 00:25:36.740231 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:36.739852 5001 scope.go:122] "RemoveContainer" containerID="ec17342524a50c2c5323dda527cdfddeb0d6c6b672677176bf4892faa6da465d" Oct 30 00:25:36.740614 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:25:36.740501 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec17342524a50c2c5323dda527cdfddeb0d6c6b672677176bf4892faa6da465d\": container with ID starting with ec17342524a50c2c5323dda527cdfddeb0d6c6b672677176bf4892faa6da465d not found: ID does not exist" containerID="ec17342524a50c2c5323dda527cdfddeb0d6c6b672677176bf4892faa6da465d" Oct 30 00:25:36.740614 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:36.740544 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec17342524a50c2c5323dda527cdfddeb0d6c6b672677176bf4892faa6da465d"} err="failed to get container status \"ec17342524a50c2c5323dda527cdfddeb0d6c6b672677176bf4892faa6da465d\": rpc error: code = NotFound desc = could not find container \"ec17342524a50c2c5323dda527cdfddeb0d6c6b672677176bf4892faa6da465d\": container with ID starting with ec17342524a50c2c5323dda527cdfddeb0d6c6b672677176bf4892faa6da465d not found: ID does not exist" Oct 30 00:25:36.761368 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:36.761337 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds"] Oct 30 00:25:36.767229 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:36.767198 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-0847e8c0/e2e-lifecycle-model-0847e8c0-kserve-749bdd769c-kgdds"] Oct 30 00:25:36.920794 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:36.920712 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7490cfd5-895e-4d7b-a842-02bf605da03f" path="/var/lib/kubelet/pods/7490cfd5-895e-4d7b-a842-02bf605da03f/volumes" Oct 30 00:25:36.921261 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:36.921244 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3e66631-9562-4bb5-b005-1fc179a8582d" path="/var/lib/kubelet/pods/b3e66631-9562-4bb5-b005-1fc179a8582d/volumes" Oct 30 00:25:36.923246 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:36.923219 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-6b65cf7999-jkkfd"] Oct 30 00:25:37.048716 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:37.048680 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95fe8ad5-qjctz"] Oct 30 00:25:37.067790 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:37.067761 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-914a49ab-x96qs"] Oct 30 00:25:37.073571 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:37.073520 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95fe8ad5-qjctz"] Oct 30 00:25:37.079264 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:37.079239 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-914a49ab-x96qs"] Oct 30 00:25:37.088659 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:37.088633 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:25:37.088784 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:37.088681 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" podUID="ed3d03e4-5550-40bc-8c00-48c3c7927fa1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:25:38.086578 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.086549 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:25:38.193542 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.193448 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-credential-socket\") pod \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " Oct 30 00:25:38.203792 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.193861 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-workload-socket\") pod \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " Oct 30 00:25:38.204231 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.204207 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istio-token\" (UniqueName: \"kubernetes.io/projected/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istio-token\") pod \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " Oct 30 00:25:38.213098 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.213067 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istio-envoy\") pod \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " Oct 30 00:25:38.218974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.194344 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-workload-socket" pod "ed3d03e4-5550-40bc-8c00-48c3c7927fa1" (UID: "ed3d03e4-5550-40bc-8c00-48c3c7927fa1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:38.218974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.196214 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-credential-socket" pod "ed3d03e4-5550-40bc-8c00-48c3c7927fa1" (UID: "ed3d03e4-5550-40bc-8c00-48c3c7927fa1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:38.218974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.207304 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istio-token" pod "ed3d03e4-5550-40bc-8c00-48c3c7927fa1" (UID: "ed3d03e4-5550-40bc-8c00-48c3c7927fa1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:38.271026 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.253690 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istiod-ca-cert\") pod \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " Oct 30 00:25:38.271026 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.253791 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-kube-api-access-fd82c\" (UniqueName: \"kubernetes.io/projected/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-kube-api-access-fd82c\") pod \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " Oct 30 00:25:38.271026 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.253867 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istio-data\") pod \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " Oct 30 00:25:38.271026 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.253929 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istio-podinfo\") pod \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " Oct 30 00:25:38.271026 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.253966 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-workload-certs\") pod \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\" (UID: \"ed3d03e4-5550-40bc-8c00-48c3c7927fa1\") " Oct 30 00:25:38.271026 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.255012 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:38.271026 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.255040 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:38.271026 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.255055 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:38.271026 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.257601 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istio-data" pod "ed3d03e4-5550-40bc-8c00-48c3c7927fa1" (UID: "ed3d03e4-5550-40bc-8c00-48c3c7927fa1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:38.271026 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.258433 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istiod-ca-cert" pod "ed3d03e4-5550-40bc-8c00-48c3c7927fa1" (UID: "ed3d03e4-5550-40bc-8c00-48c3c7927fa1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:25:38.271026 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.266075 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-workload-certs" pod "ed3d03e4-5550-40bc-8c00-48c3c7927fa1" (UID: "ed3d03e4-5550-40bc-8c00-48c3c7927fa1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:38.292944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.288585 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-kube-api-access-fd82c" pod "ed3d03e4-5550-40bc-8c00-48c3c7927fa1" (UID: "ed3d03e4-5550-40bc-8c00-48c3c7927fa1"). InnerVolumeSpecName "kube-api-access-fd82c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:38.292944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.288779 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istio-envoy" pod "ed3d03e4-5550-40bc-8c00-48c3c7927fa1" (UID: "ed3d03e4-5550-40bc-8c00-48c3c7927fa1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:38.298112 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.296804 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istio-podinfo" pod "ed3d03e4-5550-40bc-8c00-48c3c7927fa1" (UID: "ed3d03e4-5550-40bc-8c00-48c3c7927fa1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:25:38.356324 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.356244 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:38.356324 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.356270 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:38.356324 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.356283 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:38.356324 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.356296 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd82c\" (UniqueName: \"kubernetes.io/projected/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-kube-api-access-fd82c\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:38.356324 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.356308 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:38.356324 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.356322 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed3d03e4-5550-40bc-8c00-48c3c7927fa1-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:38.370458 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.370423 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a55e1-m758p"] Oct 30 00:25:38.371166 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.371150 5001 cpu_manager.go:414] "containerMap: removing container" podUID="c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8" containerName="revoke-keys" Oct 30 00:25:38.371224 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.371167 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8" containerName="revoke-keys" Oct 30 00:25:38.371224 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.371186 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4bfbc166-3dbf-4aae-96f5-515f59917846" containerName="istio-proxy" Oct 30 00:25:38.371224 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.371192 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bfbc166-3dbf-4aae-96f5-515f59917846" containerName="istio-proxy" Oct 30 00:25:38.371224 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.371204 5001 cpu_manager.go:414] "containerMap: removing container" podUID="a4253945-fb6f-438c-937e-b5061e33c84b" containerName="payload-pre-processing" Oct 30 00:25:38.371224 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.371210 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4253945-fb6f-438c-937e-b5061e33c84b" containerName="payload-pre-processing" Oct 30 00:25:38.371224 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.371221 5001 cpu_manager.go:414] "containerMap: removing container" podUID="ed3d03e4-5550-40bc-8c00-48c3c7927fa1" containerName="istio-proxy" Oct 30 00:25:38.371224 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.371226 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed3d03e4-5550-40bc-8c00-48c3c7927fa1" containerName="istio-proxy" Oct 30 00:25:38.371426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.371239 5001 cpu_manager.go:414] "containerMap: removing container" podUID="373a854c-4772-4632-b9fe-813747f39ca9" containerName="payload-processing" Oct 30 00:25:38.371426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.371244 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="373a854c-4772-4632-b9fe-813747f39ca9" containerName="payload-processing" Oct 30 00:25:38.371426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.371252 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4bfbc166-3dbf-4aae-96f5-515f59917846" containerName="istio-proxy" Oct 30 00:25:38.371426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.371257 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bfbc166-3dbf-4aae-96f5-515f59917846" containerName="istio-proxy" Oct 30 00:25:38.371426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.371265 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b3e66631-9562-4bb5-b005-1fc179a8582d" containerName="main" Oct 30 00:25:38.371426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.371269 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3e66631-9562-4bb5-b005-1fc179a8582d" containerName="main" Oct 30 00:25:38.371426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.371279 5001 cpu_manager.go:414] "containerMap: removing container" podUID="ed3d03e4-5550-40bc-8c00-48c3c7927fa1" containerName="istio-proxy" Oct 30 00:25:38.371426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.371283 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed3d03e4-5550-40bc-8c00-48c3c7927fa1" containerName="istio-proxy" Oct 30 00:25:38.371426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.371393 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8" containerName="revoke-keys" Oct 30 00:25:38.371426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.371402 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bfbc166-3dbf-4aae-96f5-515f59917846" containerName="istio-proxy" Oct 30 00:25:38.371426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.371410 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4253945-fb6f-438c-937e-b5061e33c84b" containerName="payload-pre-processing" Oct 30 00:25:38.371426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.371416 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed3d03e4-5550-40bc-8c00-48c3c7927fa1" containerName="istio-proxy" Oct 30 00:25:38.371426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.371422 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="373a854c-4772-4632-b9fe-813747f39ca9" containerName="payload-processing" Oct 30 00:25:38.371426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.371429 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bfbc166-3dbf-4aae-96f5-515f59917846" containerName="istio-proxy" Oct 30 00:25:38.371814 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.371436 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3e66631-9562-4bb5-b005-1fc179a8582d" containerName="main" Oct 30 00:25:38.375682 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.375662 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a55e1-m758p" Oct 30 00:25:38.389594 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.389567 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a55e1-m758p"] Oct 30 00:25:38.393241 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.393221 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:25:38.393370 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.393276 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:25:38.393370 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.393319 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:25:38.393554 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.393506 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:25:38.558871 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.558841 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a4b5289-1a4b-4184-a1a7-7b11b336ae56-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6a55e1-m758p\" (UID: \"7a4b5289-1a4b-4184-a1a7-7b11b336ae56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a55e1-m758p" Oct 30 00:25:38.559098 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.558966 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a4b5289-1a4b-4184-a1a7-7b11b336ae56-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6a55e1-m758p\" (UID: \"7a4b5289-1a4b-4184-a1a7-7b11b336ae56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a55e1-m758p" Oct 30 00:25:38.559098 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.559082 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpqf9\" (UniqueName: \"kubernetes.io/projected/7a4b5289-1a4b-4184-a1a7-7b11b336ae56-kube-api-access-kpqf9\") pod \"maas-api-revoke-keys-e2e-ipp-b-6a55e1-m758p\" (UID: \"7a4b5289-1a4b-4184-a1a7-7b11b336ae56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a55e1-m758p" Oct 30 00:25:38.660468 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.660369 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a4b5289-1a4b-4184-a1a7-7b11b336ae56-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6a55e1-m758p\" (UID: \"7a4b5289-1a4b-4184-a1a7-7b11b336ae56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a55e1-m758p" Oct 30 00:25:38.660643 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.660473 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpqf9\" (UniqueName: \"kubernetes.io/projected/7a4b5289-1a4b-4184-a1a7-7b11b336ae56-kube-api-access-kpqf9\") pod \"maas-api-revoke-keys-e2e-ipp-b-6a55e1-m758p\" (UID: \"7a4b5289-1a4b-4184-a1a7-7b11b336ae56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a55e1-m758p" Oct 30 00:25:38.660766 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.660748 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a4b5289-1a4b-4184-a1a7-7b11b336ae56-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6a55e1-m758p\" (UID: \"7a4b5289-1a4b-4184-a1a7-7b11b336ae56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a55e1-m758p" Oct 30 00:25:38.661178 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.661152 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a4b5289-1a4b-4184-a1a7-7b11b336ae56-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6a55e1-m758p\" (UID: \"7a4b5289-1a4b-4184-a1a7-7b11b336ae56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a55e1-m758p" Oct 30 00:25:38.663217 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.663198 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a4b5289-1a4b-4184-a1a7-7b11b336ae56-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6a55e1-m758p\" (UID: \"7a4b5289-1a4b-4184-a1a7-7b11b336ae56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a55e1-m758p" Oct 30 00:25:38.681170 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.681146 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpqf9\" (UniqueName: \"kubernetes.io/projected/7a4b5289-1a4b-4184-a1a7-7b11b336ae56-kube-api-access-kpqf9\") pod \"maas-api-revoke-keys-e2e-ipp-b-6a55e1-m758p\" (UID: \"7a4b5289-1a4b-4184-a1a7-7b11b336ae56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a55e1-m758p" Oct 30 00:25:38.705118 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.705095 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a55e1-m758p" Oct 30 00:25:38.812393 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.812365 5001 generic.go:356] "Generic (PLEG): container finished" podID="ed3d03e4-5550-40bc-8c00-48c3c7927fa1" containerID="236dd29a33f1bc6f9edc1a3037b646e0ee769237581b5c6333a209b2877f3f3f" exitCode=0 Oct 30 00:25:38.812527 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.812403 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" event={"ID":"ed3d03e4-5550-40bc-8c00-48c3c7927fa1","Type":"ContainerDied","Data":"236dd29a33f1bc6f9edc1a3037b646e0ee769237581b5c6333a209b2877f3f3f"} Oct 30 00:25:38.812527 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.812443 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" Oct 30 00:25:38.812527 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.812470 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4" event={"ID":"ed3d03e4-5550-40bc-8c00-48c3c7927fa1","Type":"ContainerDied","Data":"e0451f46023e3d7f8c60ec34a16d61c4b8a86b1035ea19c91694d8a51eef89e2"} Oct 30 00:25:38.812527 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.812501 5001 scope.go:122] "RemoveContainer" containerID="236dd29a33f1bc6f9edc1a3037b646e0ee769237581b5c6333a209b2877f3f3f" Oct 30 00:25:38.827746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.827723 5001 scope.go:122] "RemoveContainer" containerID="dc5d7805cd01f09ae7d448989b3fcd91eddda4a0896f21dd546d223c3494e577" Oct 30 00:25:38.840350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.840322 5001 scope.go:122] "RemoveContainer" containerID="236dd29a33f1bc6f9edc1a3037b646e0ee769237581b5c6333a209b2877f3f3f" Oct 30 00:25:38.840667 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:25:38.840648 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"236dd29a33f1bc6f9edc1a3037b646e0ee769237581b5c6333a209b2877f3f3f\": container with ID starting with 236dd29a33f1bc6f9edc1a3037b646e0ee769237581b5c6333a209b2877f3f3f not found: ID does not exist" containerID="236dd29a33f1bc6f9edc1a3037b646e0ee769237581b5c6333a209b2877f3f3f" Oct 30 00:25:38.840751 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.840676 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"236dd29a33f1bc6f9edc1a3037b646e0ee769237581b5c6333a209b2877f3f3f"} err="failed to get container status \"236dd29a33f1bc6f9edc1a3037b646e0ee769237581b5c6333a209b2877f3f3f\": rpc error: code = NotFound desc = could not find container \"236dd29a33f1bc6f9edc1a3037b646e0ee769237581b5c6333a209b2877f3f3f\": container with ID starting with 236dd29a33f1bc6f9edc1a3037b646e0ee769237581b5c6333a209b2877f3f3f not found: ID does not exist" Oct 30 00:25:38.840751 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.840699 5001 scope.go:122] "RemoveContainer" containerID="dc5d7805cd01f09ae7d448989b3fcd91eddda4a0896f21dd546d223c3494e577" Oct 30 00:25:38.840993 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:25:38.840969 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc5d7805cd01f09ae7d448989b3fcd91eddda4a0896f21dd546d223c3494e577\": container with ID starting with dc5d7805cd01f09ae7d448989b3fcd91eddda4a0896f21dd546d223c3494e577 not found: ID does not exist" containerID="dc5d7805cd01f09ae7d448989b3fcd91eddda4a0896f21dd546d223c3494e577" Oct 30 00:25:38.841037 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.840997 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc5d7805cd01f09ae7d448989b3fcd91eddda4a0896f21dd546d223c3494e577"} err="failed to get container status \"dc5d7805cd01f09ae7d448989b3fcd91eddda4a0896f21dd546d223c3494e577\": rpc error: code = NotFound desc = could not find container \"dc5d7805cd01f09ae7d448989b3fcd91eddda4a0896f21dd546d223c3494e577\": container with ID starting with dc5d7805cd01f09ae7d448989b3fcd91eddda4a0896f21dd546d223c3494e577 not found: ID does not exist" Oct 30 00:25:38.863400 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.863367 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4"] Oct 30 00:25:38.869476 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.869451 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f52ea1-openshift-default-756f7d574c-ns2k4"] Oct 30 00:25:38.920985 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.920897 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cad7150c-93d8-4200-abbb-a50f1d7449a2" path="/var/lib/kubelet/pods/cad7150c-93d8-4200-abbb-a50f1d7449a2/volumes" Oct 30 00:25:38.921549 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.921510 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e23c62e9-023c-4ff6-9c35-b724da9a824d" path="/var/lib/kubelet/pods/e23c62e9-023c-4ff6-9c35-b724da9a824d/volumes" Oct 30 00:25:38.922091 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:38.922072 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed3d03e4-5550-40bc-8c00-48c3c7927fa1" path="/var/lib/kubelet/pods/ed3d03e4-5550-40bc-8c00-48c3c7927fa1/volumes" Oct 30 00:25:39.011319 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:25:39.011283 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a4b5289_1a4b_4184_a1a7_7b11b336ae56.slice/crio-a70688f114f12881a9af865ea939adf4b3d238862490d725a4d3f1b8f95d5e38 WatchSource:0}: Error finding container a70688f114f12881a9af865ea939adf4b3d238862490d725a4d3f1b8f95d5e38: Status 404 returned error can't find the container with id a70688f114f12881a9af865ea939adf4b3d238862490d725a4d3f1b8f95d5e38 Oct 30 00:25:39.012387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:39.012360 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a55e1-m758p"] Oct 30 00:25:39.919462 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:39.919422 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a55e1-m758p" event={"ID":"7a4b5289-1a4b-4184-a1a7-7b11b336ae56","Type":"ContainerStarted","Data":"6a446e0894090fcc46a51bb6a1dc982d1af2dfcf9055f72ea27ab61e444f4abd"} Oct 30 00:25:39.920026 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:39.920001 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a55e1-m758p" event={"ID":"7a4b5289-1a4b-4184-a1a7-7b11b336ae56","Type":"ContainerStarted","Data":"a70688f114f12881a9af865ea939adf4b3d238862490d725a4d3f1b8f95d5e38"} Oct 30 00:25:39.974441 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:39.968003 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a55e1-m758p" podStartSLOduration=1.9679841740000001 podStartE2EDuration="1.967984174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:25:39.966253608 +0000 UTC m=+1255.171754789" watchObservedRunningTime="2026-10-30 00:25:39.967984174 +0000 UTC m=+1255.173485359" Oct 30 00:25:40.189164 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:40.189135 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk"] Oct 30 00:25:40.342951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:40.341077 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0aea47-dhcvk"] Oct 30 00:25:40.920805 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:40.920770 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1da01660-8624-4707-bd34-edba0f103f2f" path="/var/lib/kubelet/pods/1da01660-8624-4707-bd34-edba0f103f2f/volumes" Oct 30 00:25:40.929690 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:40.929665 5001 generic.go:356] "Generic (PLEG): container finished" podID="7a4b5289-1a4b-4184-a1a7-7b11b336ae56" containerID="6a446e0894090fcc46a51bb6a1dc982d1af2dfcf9055f72ea27ab61e444f4abd" exitCode=0 Oct 30 00:25:40.929826 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:40.929754 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a55e1-m758p" event={"ID":"7a4b5289-1a4b-4184-a1a7-7b11b336ae56","Type":"ContainerDied","Data":"6a446e0894090fcc46a51bb6a1dc982d1af2dfcf9055f72ea27ab61e444f4abd"} Oct 30 00:25:41.022076 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:41.022045 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-7f5fd784c9-pjrvt"] Oct 30 00:25:41.923713 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:41.923669 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-6b65cf7999-jkkfd" podUID="f91ba282-c730-4f93-ba5b-92a63bad3488" containerName="payload-pre-processing" containerID="cri-o://904f1d651fadd6bd68979e5892f4c6eec717598a47c684f82b6c24b7428d0ec4" gracePeriod=25 Oct 30 00:25:42.374730 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:42.374704 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a55e1-m758p" Oct 30 00:25:42.378389 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:42.378367 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-6b65cf7999-jkkfd" Oct 30 00:25:42.553120 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:42.553083 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a4b5289-1a4b-4184-a1a7-7b11b336ae56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a4b5289-1a4b-4184-a1a7-7b11b336ae56-maas-api-openshift-ca\") pod \"7a4b5289-1a4b-4184-a1a7-7b11b336ae56\" (UID: \"7a4b5289-1a4b-4184-a1a7-7b11b336ae56\") " Oct 30 00:25:42.553305 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:42.553163 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f91ba282-c730-4f93-ba5b-92a63bad3488-kube-api-access-rbdnw\" (UniqueName: \"kubernetes.io/projected/f91ba282-c730-4f93-ba5b-92a63bad3488-kube-api-access-rbdnw\") pod \"f91ba282-c730-4f93-ba5b-92a63bad3488\" (UID: \"f91ba282-c730-4f93-ba5b-92a63bad3488\") " Oct 30 00:25:42.553305 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:42.553199 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a4b5289-1a4b-4184-a1a7-7b11b336ae56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a4b5289-1a4b-4184-a1a7-7b11b336ae56-maas-api-xks-ca\") pod \"7a4b5289-1a4b-4184-a1a7-7b11b336ae56\" (UID: \"7a4b5289-1a4b-4184-a1a7-7b11b336ae56\") " Oct 30 00:25:42.553305 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:42.553277 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a4b5289-1a4b-4184-a1a7-7b11b336ae56-kube-api-access-kpqf9\" (UniqueName: \"kubernetes.io/projected/7a4b5289-1a4b-4184-a1a7-7b11b336ae56-kube-api-access-kpqf9\") pod \"7a4b5289-1a4b-4184-a1a7-7b11b336ae56\" (UID: \"7a4b5289-1a4b-4184-a1a7-7b11b336ae56\") " Oct 30 00:25:42.553436 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:42.553385 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f91ba282-c730-4f93-ba5b-92a63bad3488-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f91ba282-c730-4f93-ba5b-92a63bad3488-plugins-config-volume\") pod \"f91ba282-c730-4f93-ba5b-92a63bad3488\" (UID: \"f91ba282-c730-4f93-ba5b-92a63bad3488\") " Oct 30 00:25:42.553486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:42.553452 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a4b5289-1a4b-4184-a1a7-7b11b336ae56-maas-api-openshift-ca" pod "7a4b5289-1a4b-4184-a1a7-7b11b336ae56" (UID: "7a4b5289-1a4b-4184-a1a7-7b11b336ae56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:25:42.553846 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:42.553816 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f91ba282-c730-4f93-ba5b-92a63bad3488-plugins-config-volume" pod "f91ba282-c730-4f93-ba5b-92a63bad3488" (UID: "f91ba282-c730-4f93-ba5b-92a63bad3488"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:25:42.554169 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:42.554145 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f91ba282-c730-4f93-ba5b-92a63bad3488-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:42.554264 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:42.554171 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a4b5289-1a4b-4184-a1a7-7b11b336ae56-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:42.555403 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:42.555379 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a4b5289-1a4b-4184-a1a7-7b11b336ae56-kube-api-access-kpqf9" pod "7a4b5289-1a4b-4184-a1a7-7b11b336ae56" (UID: "7a4b5289-1a4b-4184-a1a7-7b11b336ae56"). InnerVolumeSpecName "kube-api-access-kpqf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:42.555403 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:42.555388 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a4b5289-1a4b-4184-a1a7-7b11b336ae56-maas-api-xks-ca" pod "7a4b5289-1a4b-4184-a1a7-7b11b336ae56" (UID: "7a4b5289-1a4b-4184-a1a7-7b11b336ae56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:25:42.555554 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:42.555412 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f91ba282-c730-4f93-ba5b-92a63bad3488-kube-api-access-rbdnw" pod "f91ba282-c730-4f93-ba5b-92a63bad3488" (UID: "f91ba282-c730-4f93-ba5b-92a63bad3488"). InnerVolumeSpecName "kube-api-access-rbdnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:42.655250 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:42.655219 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbdnw\" (UniqueName: \"kubernetes.io/projected/f91ba282-c730-4f93-ba5b-92a63bad3488-kube-api-access-rbdnw\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:42.655250 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:42.655248 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a4b5289-1a4b-4184-a1a7-7b11b336ae56-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:42.655497 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:42.655261 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpqf9\" (UniqueName: \"kubernetes.io/projected/7a4b5289-1a4b-4184-a1a7-7b11b336ae56-kube-api-access-kpqf9\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:42.996285 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:42.996255 5001 generic.go:356] "Generic (PLEG): container finished" podID="f91ba282-c730-4f93-ba5b-92a63bad3488" containerID="904f1d651fadd6bd68979e5892f4c6eec717598a47c684f82b6c24b7428d0ec4" exitCode=0 Oct 30 00:25:42.996728 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:42.996305 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-6b65cf7999-jkkfd" Oct 30 00:25:42.996728 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:42.996335 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-6b65cf7999-jkkfd" event={"ID":"f91ba282-c730-4f93-ba5b-92a63bad3488","Type":"ContainerDied","Data":"904f1d651fadd6bd68979e5892f4c6eec717598a47c684f82b6c24b7428d0ec4"} Oct 30 00:25:42.996728 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:42.996386 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-6b65cf7999-jkkfd" event={"ID":"f91ba282-c730-4f93-ba5b-92a63bad3488","Type":"ContainerDied","Data":"04253af86b85c37a22077493f7a7733a30ca53cf26f384ea3c1b692e790460ae"} Oct 30 00:25:42.996728 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:42.996408 5001 scope.go:122] "RemoveContainer" containerID="904f1d651fadd6bd68979e5892f4c6eec717598a47c684f82b6c24b7428d0ec4" Oct 30 00:25:42.998451 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:42.998317 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a55e1-m758p" event={"ID":"7a4b5289-1a4b-4184-a1a7-7b11b336ae56","Type":"ContainerDied","Data":"a70688f114f12881a9af865ea939adf4b3d238862490d725a4d3f1b8f95d5e38"} Oct 30 00:25:42.998451 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:42.998331 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a55e1-m758p" Oct 30 00:25:42.998451 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:42.998361 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a70688f114f12881a9af865ea939adf4b3d238862490d725a4d3f1b8f95d5e38" Oct 30 00:25:43.008157 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:43.008133 5001 scope.go:122] "RemoveContainer" containerID="904f1d651fadd6bd68979e5892f4c6eec717598a47c684f82b6c24b7428d0ec4" Oct 30 00:25:43.008381 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:25:43.008362 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"904f1d651fadd6bd68979e5892f4c6eec717598a47c684f82b6c24b7428d0ec4\": container with ID starting with 904f1d651fadd6bd68979e5892f4c6eec717598a47c684f82b6c24b7428d0ec4 not found: ID does not exist" containerID="904f1d651fadd6bd68979e5892f4c6eec717598a47c684f82b6c24b7428d0ec4" Oct 30 00:25:43.008454 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:43.008387 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"904f1d651fadd6bd68979e5892f4c6eec717598a47c684f82b6c24b7428d0ec4"} err="failed to get container status \"904f1d651fadd6bd68979e5892f4c6eec717598a47c684f82b6c24b7428d0ec4\": rpc error: code = NotFound desc = could not find container \"904f1d651fadd6bd68979e5892f4c6eec717598a47c684f82b6c24b7428d0ec4\": container with ID starting with 904f1d651fadd6bd68979e5892f4c6eec717598a47c684f82b6c24b7428d0ec4 not found: ID does not exist" Oct 30 00:25:43.054946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:43.054906 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-6b65cf7999-jkkfd"] Oct 30 00:25:43.064780 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:43.064751 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-6b65cf7999-jkkfd"] Oct 30 00:25:44.933119 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:44.933083 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f91ba282-c730-4f93-ba5b-92a63bad3488" path="/var/lib/kubelet/pods/f91ba282-c730-4f93-ba5b-92a63bad3488/volumes" Oct 30 00:25:46.022896 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:46.022845 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-7f5fd784c9-pjrvt" podUID="259e6702-4256-4bbe-b2e0-86b4d2071f66" containerName="payload-pre-processing" containerID="cri-o://1a049553580afe928257072be796eaeb02b25aa307040fbfc686923d646557a8" gracePeriod=25 Oct 30 00:25:46.337046 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:46.337019 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-7f5fd784c9-pjrvt" Oct 30 00:25:46.419003 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:46.416905 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/259e6702-4256-4bbe-b2e0-86b4d2071f66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259e6702-4256-4bbe-b2e0-86b4d2071f66-plugins-config-volume\") pod \"259e6702-4256-4bbe-b2e0-86b4d2071f66\" (UID: \"259e6702-4256-4bbe-b2e0-86b4d2071f66\") " Oct 30 00:25:46.419003 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:46.417039 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/259e6702-4256-4bbe-b2e0-86b4d2071f66-kube-api-access-rrg85\" (UniqueName: \"kubernetes.io/projected/259e6702-4256-4bbe-b2e0-86b4d2071f66-kube-api-access-rrg85\") pod \"259e6702-4256-4bbe-b2e0-86b4d2071f66\" (UID: \"259e6702-4256-4bbe-b2e0-86b4d2071f66\") " Oct 30 00:25:46.419003 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:46.418692 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/259e6702-4256-4bbe-b2e0-86b4d2071f66-plugins-config-volume" pod "259e6702-4256-4bbe-b2e0-86b4d2071f66" (UID: "259e6702-4256-4bbe-b2e0-86b4d2071f66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:25:46.421712 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:46.421679 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/259e6702-4256-4bbe-b2e0-86b4d2071f66-kube-api-access-rrg85" pod "259e6702-4256-4bbe-b2e0-86b4d2071f66" (UID: "259e6702-4256-4bbe-b2e0-86b4d2071f66"). InnerVolumeSpecName "kube-api-access-rrg85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:46.519560 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:46.519527 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrg85\" (UniqueName: \"kubernetes.io/projected/259e6702-4256-4bbe-b2e0-86b4d2071f66-kube-api-access-rrg85\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:46.519560 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:46.519554 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259e6702-4256-4bbe-b2e0-86b4d2071f66-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:47.048029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:47.047994 5001 generic.go:356] "Generic (PLEG): container finished" podID="259e6702-4256-4bbe-b2e0-86b4d2071f66" containerID="1a049553580afe928257072be796eaeb02b25aa307040fbfc686923d646557a8" exitCode=0 Oct 30 00:25:47.048506 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:47.048180 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-7f5fd784c9-pjrvt" event={"ID":"259e6702-4256-4bbe-b2e0-86b4d2071f66","Type":"ContainerDied","Data":"1a049553580afe928257072be796eaeb02b25aa307040fbfc686923d646557a8"} Oct 30 00:25:47.048506 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:47.048218 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-7f5fd784c9-pjrvt" event={"ID":"259e6702-4256-4bbe-b2e0-86b4d2071f66","Type":"ContainerDied","Data":"272a6bef0307e8c6b1581bf27be8106c398e781bb295f920e58bf6835f4809c9"} Oct 30 00:25:47.048506 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:47.048239 5001 scope.go:122] "RemoveContainer" containerID="1a049553580afe928257072be796eaeb02b25aa307040fbfc686923d646557a8" Oct 30 00:25:47.048506 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:47.048371 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-7f5fd784c9-pjrvt" Oct 30 00:25:47.060618 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:47.060583 5001 scope.go:122] "RemoveContainer" containerID="1a049553580afe928257072be796eaeb02b25aa307040fbfc686923d646557a8" Oct 30 00:25:47.060956 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:25:47.060929 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a049553580afe928257072be796eaeb02b25aa307040fbfc686923d646557a8\": container with ID starting with 1a049553580afe928257072be796eaeb02b25aa307040fbfc686923d646557a8 not found: ID does not exist" containerID="1a049553580afe928257072be796eaeb02b25aa307040fbfc686923d646557a8" Oct 30 00:25:47.061077 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:47.060964 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a049553580afe928257072be796eaeb02b25aa307040fbfc686923d646557a8"} err="failed to get container status \"1a049553580afe928257072be796eaeb02b25aa307040fbfc686923d646557a8\": rpc error: code = NotFound desc = could not find container \"1a049553580afe928257072be796eaeb02b25aa307040fbfc686923d646557a8\": container with ID starting with 1a049553580afe928257072be796eaeb02b25aa307040fbfc686923d646557a8 not found: ID does not exist" Oct 30 00:25:47.097880 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:47.097844 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-7f5fd784c9-pjrvt"] Oct 30 00:25:47.106902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:47.106874 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb3ad-7f5fd784c9-pjrvt"] Oct 30 00:25:47.174102 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:47.174001 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g"] Oct 30 00:25:47.174285 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:47.174252 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" podUID="a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7" containerName="istio-proxy" containerID="cri-o://d6025520905dea45a36fdf8a8ec0387ffbff91650d87c4bb7af7c87b2e06524c" gracePeriod=30 Oct 30 00:25:47.566932 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:25:47.562137 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:25:48.920009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:48.919978 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="259e6702-4256-4bbe-b2e0-86b4d2071f66" path="/var/lib/kubelet/pods/259e6702-4256-4bbe-b2e0-86b4d2071f66/volumes" Oct 30 00:25:51.333933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:51.329499 5001 scope.go:122] "RemoveContainer" containerID="8306866e7c1dfd59aa7470424ce9271581eb1a209a96f3c8c0192ef19a4e246f" Oct 30 00:25:51.439751 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:51.439717 5001 scope.go:122] "RemoveContainer" containerID="8c4e23d5a1341eda01baffa2b857efb5ce53b62e4c79e483ec75d4fe2e59a49b" Oct 30 00:25:51.460321 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:51.459892 5001 scope.go:122] "RemoveContainer" containerID="5e7f4e46b9ea4a2f89e35ca15fbbe81fcc34b6794fb39c937af2cebc4610bd97" Oct 30 00:25:51.549352 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:51.549324 5001 scope.go:122] "RemoveContainer" containerID="b726d60724c98a99053d6dfcedf5a7304e311c70a9240f3e70945dc3684d41d8" Oct 30 00:25:51.630144 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:51.630110 5001 scope.go:122] "RemoveContainer" containerID="3c0688d39574f44f040ef9e0d45573a3de277ca220435ba8d8f80927cebaf6ba" Oct 30 00:25:51.707291 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:51.707255 5001 scope.go:122] "RemoveContainer" containerID="7df666dee96e08d14d85fb3df3b65106261c97198e55703225d211e91b144aa8" Oct 30 00:25:51.797371 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:51.797344 5001 scope.go:122] "RemoveContainer" containerID="94cd6aa26317edd3d8e0e017c599b9a628fb9e960436c76b7932a3f04ed77406" Oct 30 00:25:52.042902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:52.042826 5001 patch_prober.go:29] interesting pod/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:25:52.042902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:52.042893 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" podUID="a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:25:52.049032 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:52.049000 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5368f66-fltv9"] Oct 30 00:25:52.054448 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:52.054424 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5368f66-fltv9"] Oct 30 00:25:52.605688 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:52.605665 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:25:52.708271 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:52.707569 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-workload-certs\") pod \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " Oct 30 00:25:52.708271 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:52.707651 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istiod-ca-cert\") pod \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " Oct 30 00:25:52.708271 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:52.707693 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-credential-socket\") pod \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " Oct 30 00:25:52.708271 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:52.707810 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-kube-api-access-rbxmg\" (UniqueName: \"kubernetes.io/projected/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-kube-api-access-rbxmg\") pod \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " Oct 30 00:25:52.708271 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:52.707848 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istio-data\") pod \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " Oct 30 00:25:52.708271 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:52.707895 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-workload-certs" pod "a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7" (UID: "a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:52.713458 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:52.707905 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istio-token\" (UniqueName: \"kubernetes.io/projected/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istio-token\") pod \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " Oct 30 00:25:52.713458 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:52.712055 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istio-podinfo\") pod \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " Oct 30 00:25:52.713458 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:52.712091 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istiod-ca-cert" pod "a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7" (UID: "a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:25:52.713458 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:52.712264 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-workload-socket\") pod \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " Oct 30 00:25:52.713458 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:52.712342 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istio-envoy\") pod \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\" (UID: \"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7\") " Oct 30 00:25:52.713458 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:52.713201 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:52.713458 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:52.713224 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:52.713458 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:52.713314 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-workload-socket" pod "a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7" (UID: "a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:52.713458 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:52.713379 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-credential-socket" pod "a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7" (UID: "a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:52.713458 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:52.713432 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istio-data" pod "a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7" (UID: "a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:52.725935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:52.723093 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istio-token" pod "a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7" (UID: "a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:52.735979 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:52.731211 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istio-podinfo" pod "a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7" (UID: "a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:25:52.736739 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:52.736711 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-kube-api-access-rbxmg" pod "a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7" (UID: "a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7"). InnerVolumeSpecName "kube-api-access-rbxmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:52.747927 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:52.745004 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istio-envoy" pod "a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7" (UID: "a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:25:52.817934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:52.816321 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:52.817934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:52.816363 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:52.817934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:52.816379 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:52.817934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:52.816394 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:52.817934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:52.816408 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:52.817934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:52.816426 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbxmg\" (UniqueName: \"kubernetes.io/projected/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-kube-api-access-rbxmg\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:52.817934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:52.816440 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:52.965985 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:52.959657 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="628e474e-57df-4912-a13c-9c83ab3b0086" path="/var/lib/kubelet/pods/628e474e-57df-4912-a13c-9c83ab3b0086/volumes" Oct 30 00:25:53.185982 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:53.185265 5001 generic.go:356] "Generic (PLEG): container finished" podID="a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7" containerID="d6025520905dea45a36fdf8a8ec0387ffbff91650d87c4bb7af7c87b2e06524c" exitCode=0 Oct 30 00:25:53.185982 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:53.185501 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" event={"ID":"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7","Type":"ContainerDied","Data":"d6025520905dea45a36fdf8a8ec0387ffbff91650d87c4bb7af7c87b2e06524c"} Oct 30 00:25:53.185982 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:53.185557 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" event={"ID":"a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7","Type":"ContainerDied","Data":"194adfb9bd691a607cfd09475c3511d9cc32fef82d73cc9c1fca6aa7e4741aef"} Oct 30 00:25:53.185982 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:53.185590 5001 scope.go:122] "RemoveContainer" containerID="d6025520905dea45a36fdf8a8ec0387ffbff91650d87c4bb7af7c87b2e06524c" Oct 30 00:25:53.185982 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:53.185749 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g" Oct 30 00:25:53.230214 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:53.226105 5001 scope.go:122] "RemoveContainer" containerID="d8349c31038fd354f846c7f57f925a6d777d8112a1b5b84ce9d9c75c0a5f42ad" Oct 30 00:25:53.267252 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:53.266107 5001 scope.go:122] "RemoveContainer" containerID="d6025520905dea45a36fdf8a8ec0387ffbff91650d87c4bb7af7c87b2e06524c" Oct 30 00:25:53.267252 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:25:53.266885 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6025520905dea45a36fdf8a8ec0387ffbff91650d87c4bb7af7c87b2e06524c\": container with ID starting with d6025520905dea45a36fdf8a8ec0387ffbff91650d87c4bb7af7c87b2e06524c not found: ID does not exist" containerID="d6025520905dea45a36fdf8a8ec0387ffbff91650d87c4bb7af7c87b2e06524c" Oct 30 00:25:53.267252 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:53.266955 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6025520905dea45a36fdf8a8ec0387ffbff91650d87c4bb7af7c87b2e06524c"} err="failed to get container status \"d6025520905dea45a36fdf8a8ec0387ffbff91650d87c4bb7af7c87b2e06524c\": rpc error: code = NotFound desc = could not find container \"d6025520905dea45a36fdf8a8ec0387ffbff91650d87c4bb7af7c87b2e06524c\": container with ID starting with d6025520905dea45a36fdf8a8ec0387ffbff91650d87c4bb7af7c87b2e06524c not found: ID does not exist" Oct 30 00:25:53.267252 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:53.266980 5001 scope.go:122] "RemoveContainer" containerID="d8349c31038fd354f846c7f57f925a6d777d8112a1b5b84ce9d9c75c0a5f42ad" Oct 30 00:25:53.267741 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:25:53.267557 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8349c31038fd354f846c7f57f925a6d777d8112a1b5b84ce9d9c75c0a5f42ad\": container with ID starting with d8349c31038fd354f846c7f57f925a6d777d8112a1b5b84ce9d9c75c0a5f42ad not found: ID does not exist" containerID="d8349c31038fd354f846c7f57f925a6d777d8112a1b5b84ce9d9c75c0a5f42ad" Oct 30 00:25:53.267741 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:53.267606 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8349c31038fd354f846c7f57f925a6d777d8112a1b5b84ce9d9c75c0a5f42ad"} err="failed to get container status \"d8349c31038fd354f846c7f57f925a6d777d8112a1b5b84ce9d9c75c0a5f42ad\": rpc error: code = NotFound desc = could not find container \"d8349c31038fd354f846c7f57f925a6d777d8112a1b5b84ce9d9c75c0a5f42ad\": container with ID starting with d8349c31038fd354f846c7f57f925a6d777d8112a1b5b84ce9d9c75c0a5f42ad not found: ID does not exist" Oct 30 00:25:53.269976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:53.269953 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g"] Oct 30 00:25:53.274426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:53.274401 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-0aea47-openshift-default-b7745f5d7-hjt6g"] Oct 30 00:25:55.310371 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:55.310329 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7" path="/var/lib/kubelet/pods/a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7/volumes" Oct 30 00:25:55.321610 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:55.321576 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78"] Oct 30 00:25:55.321852 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:55.321837 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08dd33-zzt78"] Oct 30 00:25:57.025785 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:57.025749 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccde7ce6-b81a-4564-a8f2-15669279f5a9" path="/var/lib/kubelet/pods/ccde7ce6-b81a-4564-a8f2-15669279f5a9/volumes" Oct 30 00:25:58.124047 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:58.123949 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-7b5c564646-k975p"] Oct 30 00:25:58.124859 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:58.124819 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-7b5c564646-k975p" podUID="badf6bc7-f9da-431f-8007-9f7b53cec3a1" containerName="payload-processing" containerID="cri-o://460666f4942a66c66b96715d4dde6993258d701de591a843b16bd524dbf2d3a0" gracePeriod=30 Oct 30 00:25:58.146881 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:58.146833 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-84dc9c97ff-6tsx5"] Oct 30 00:25:58.147126 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:58.147072 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-84dc9c97ff-6tsx5" podUID="de1c480c-7dbc-47b0-bdff-d9accfbb0944" containerName="payload-pre-processing" containerID="cri-o://ba9a6745f35e9b36e6ad069a8ed0b7c0b28dedde8d719dd28e990afd7bb5f2ed" gracePeriod=30 Oct 30 00:25:58.459529 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:58.459199 5001 generic.go:356] "Generic (PLEG): container finished" podID="de1c480c-7dbc-47b0-bdff-d9accfbb0944" containerID="ba9a6745f35e9b36e6ad069a8ed0b7c0b28dedde8d719dd28e990afd7bb5f2ed" exitCode=0 Oct 30 00:25:58.459529 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:58.459468 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-84dc9c97ff-6tsx5" event={"ID":"de1c480c-7dbc-47b0-bdff-d9accfbb0944","Type":"ContainerDied","Data":"ba9a6745f35e9b36e6ad069a8ed0b7c0b28dedde8d719dd28e990afd7bb5f2ed"} Oct 30 00:25:58.462334 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:58.462117 5001 generic.go:356] "Generic (PLEG): container finished" podID="badf6bc7-f9da-431f-8007-9f7b53cec3a1" containerID="460666f4942a66c66b96715d4dde6993258d701de591a843b16bd524dbf2d3a0" exitCode=0 Oct 30 00:25:58.462334 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:58.462283 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-7b5c564646-k975p" event={"ID":"badf6bc7-f9da-431f-8007-9f7b53cec3a1","Type":"ContainerDied","Data":"460666f4942a66c66b96715d4dde6993258d701de591a843b16bd524dbf2d3a0"} Oct 30 00:25:58.495016 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:58.494415 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96027e-76ddd5685d-v4sqk"] Oct 30 00:25:58.495533 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:58.495496 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-76ddd5685d-v4sqk" podUID="c6ba9acb-5f2f-4028-9963-fe2201dccffe" containerName="payload-processing" containerID="cri-o://382793f6756a34a19463fb58d36ff50b5582b757a1fe27543ce80e7fe3c67dcb" gracePeriod=30 Oct 30 00:25:58.666014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:58.658325 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a55e1-m758p"] Oct 30 00:25:58.689982 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:58.672492 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-7b5c564646-k975p" Oct 30 00:25:58.691581 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:58.691551 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a55e1-m758p"] Oct 30 00:25:58.797937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:58.788752 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/badf6bc7-f9da-431f-8007-9f7b53cec3a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/badf6bc7-f9da-431f-8007-9f7b53cec3a1-plugins-config-volume\") pod \"badf6bc7-f9da-431f-8007-9f7b53cec3a1\" (UID: \"badf6bc7-f9da-431f-8007-9f7b53cec3a1\") " Oct 30 00:25:58.797937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:58.789131 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/badf6bc7-f9da-431f-8007-9f7b53cec3a1-kube-api-access-5r6pf\" (UniqueName: \"kubernetes.io/projected/badf6bc7-f9da-431f-8007-9f7b53cec3a1-kube-api-access-5r6pf\") pod \"badf6bc7-f9da-431f-8007-9f7b53cec3a1\" (UID: \"badf6bc7-f9da-431f-8007-9f7b53cec3a1\") " Oct 30 00:25:58.799389 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:58.798996 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/badf6bc7-f9da-431f-8007-9f7b53cec3a1-plugins-config-volume" pod "badf6bc7-f9da-431f-8007-9f7b53cec3a1" (UID: "badf6bc7-f9da-431f-8007-9f7b53cec3a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:25:58.850935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:58.807065 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-84dc9c97ff-6tsx5" Oct 30 00:25:58.910938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:58.909649 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de1c480c-7dbc-47b0-bdff-d9accfbb0944-kube-api-access-q2nkw\" (UniqueName: \"kubernetes.io/projected/de1c480c-7dbc-47b0-bdff-d9accfbb0944-kube-api-access-q2nkw\") pod \"de1c480c-7dbc-47b0-bdff-d9accfbb0944\" (UID: \"de1c480c-7dbc-47b0-bdff-d9accfbb0944\") " Oct 30 00:25:58.910938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:58.909781 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de1c480c-7dbc-47b0-bdff-d9accfbb0944-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de1c480c-7dbc-47b0-bdff-d9accfbb0944-plugins-config-volume\") pod \"de1c480c-7dbc-47b0-bdff-d9accfbb0944\" (UID: \"de1c480c-7dbc-47b0-bdff-d9accfbb0944\") " Oct 30 00:25:58.910938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:58.910619 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/badf6bc7-f9da-431f-8007-9f7b53cec3a1-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:58.983997 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:58.978051 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de1c480c-7dbc-47b0-bdff-d9accfbb0944-plugins-config-volume" pod "de1c480c-7dbc-47b0-bdff-d9accfbb0944" (UID: "de1c480c-7dbc-47b0-bdff-d9accfbb0944"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:25:59.029943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:59.015530 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de1c480c-7dbc-47b0-bdff-d9accfbb0944-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:59.070213 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:59.066805 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de1c480c-7dbc-47b0-bdff-d9accfbb0944-kube-api-access-q2nkw" pod "de1c480c-7dbc-47b0-bdff-d9accfbb0944" (UID: "de1c480c-7dbc-47b0-bdff-d9accfbb0944"). InnerVolumeSpecName "kube-api-access-q2nkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:59.081980 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:59.081856 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/badf6bc7-f9da-431f-8007-9f7b53cec3a1-kube-api-access-5r6pf" pod "badf6bc7-f9da-431f-8007-9f7b53cec3a1" (UID: "badf6bc7-f9da-431f-8007-9f7b53cec3a1"). InnerVolumeSpecName "kube-api-access-5r6pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:25:59.087544 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:59.087511 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a4b5289-1a4b-4184-a1a7-7b11b336ae56" path="/var/lib/kubelet/pods/7a4b5289-1a4b-4184-a1a7-7b11b336ae56/volumes" Oct 30 00:25:59.175860 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:59.127390 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r6pf\" (UniqueName: \"kubernetes.io/projected/badf6bc7-f9da-431f-8007-9f7b53cec3a1-kube-api-access-5r6pf\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:59.175860 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:59.127430 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2nkw\" (UniqueName: \"kubernetes.io/projected/de1c480c-7dbc-47b0-bdff-d9accfbb0944-kube-api-access-q2nkw\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:25:59.230739 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:59.230681 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96027e-gn8cz"] Oct 30 00:25:59.444937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:59.420873 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96027e-gn8cz"] Oct 30 00:25:59.674586 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:59.674486 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-7b5c564646-k975p" podUID="badf6bc7-f9da-431f-8007-9f7b53cec3a1" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.167:9004: i/o timeout" Oct 30 00:25:59.726331 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:59.689011 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-84dc9c97ff-6tsx5" podUID="de1c480c-7dbc-47b0-bdff-d9accfbb0944" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.166:9004: i/o timeout" Oct 30 00:25:59.764330 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:59.764290 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-84dc9c97ff-6tsx5" event={"ID":"de1c480c-7dbc-47b0-bdff-d9accfbb0944","Type":"ContainerDied","Data":"3d24754752fc618067144152fb654fabc31e77d060ab6160b7ec9e7cbd10dc92"} Oct 30 00:25:59.764596 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:59.764578 5001 scope.go:122] "RemoveContainer" containerID="ba9a6745f35e9b36e6ad069a8ed0b7c0b28dedde8d719dd28e990afd7bb5f2ed" Oct 30 00:25:59.765887 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:59.764852 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-84dc9c97ff-6tsx5" Oct 30 00:25:59.775823 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:59.775779 5001 generic.go:356] "Generic (PLEG): container finished" podID="c6ba9acb-5f2f-4028-9963-fe2201dccffe" containerID="382793f6756a34a19463fb58d36ff50b5582b757a1fe27543ce80e7fe3c67dcb" exitCode=0 Oct 30 00:25:59.776219 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:59.776159 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-76ddd5685d-v4sqk" event={"ID":"c6ba9acb-5f2f-4028-9963-fe2201dccffe","Type":"ContainerDied","Data":"382793f6756a34a19463fb58d36ff50b5582b757a1fe27543ce80e7fe3c67dcb"} Oct 30 00:25:59.796955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:59.796357 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-7b5c564646-k975p" event={"ID":"badf6bc7-f9da-431f-8007-9f7b53cec3a1","Type":"ContainerDied","Data":"aaf4021b852b597cfec0b2b8b5dadeed199788b72ce19afd2a6be3466876e3bf"} Oct 30 00:25:59.796955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:59.796472 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-7b5c564646-k975p" Oct 30 00:25:59.907313 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:59.904232 5001 scope.go:122] "RemoveContainer" containerID="460666f4942a66c66b96715d4dde6993258d701de591a843b16bd524dbf2d3a0" Oct 30 00:25:59.953866 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:59.953198 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-76ddd5685d-v4sqk" Oct 30 00:25:59.962846 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:59.962819 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-84dc9c97ff-6tsx5"] Oct 30 00:25:59.972798 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:25:59.972760 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a55e1-84dc9c97ff-6tsx5"] Oct 30 00:26:00.008824 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:00.008792 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6ba9acb-5f2f-4028-9963-fe2201dccffe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6ba9acb-5f2f-4028-9963-fe2201dccffe-plugins-config-volume\") pod \"c6ba9acb-5f2f-4028-9963-fe2201dccffe\" (UID: \"c6ba9acb-5f2f-4028-9963-fe2201dccffe\") " Oct 30 00:26:00.009632 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:00.009488 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6ba9acb-5f2f-4028-9963-fe2201dccffe-plugins-config-volume" pod "c6ba9acb-5f2f-4028-9963-fe2201dccffe" (UID: "c6ba9acb-5f2f-4028-9963-fe2201dccffe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:26:00.009884 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:00.009863 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6ba9acb-5f2f-4028-9963-fe2201dccffe-kube-api-access-lnptb\" (UniqueName: \"kubernetes.io/projected/c6ba9acb-5f2f-4028-9963-fe2201dccffe-kube-api-access-lnptb\") pod \"c6ba9acb-5f2f-4028-9963-fe2201dccffe\" (UID: \"c6ba9acb-5f2f-4028-9963-fe2201dccffe\") " Oct 30 00:26:00.012691 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:00.012668 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6ba9acb-5f2f-4028-9963-fe2201dccffe-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:26:00.019205 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:00.019175 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-7b5c564646-k975p"] Oct 30 00:26:00.070260 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:00.070186 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ba9acb-5f2f-4028-9963-fe2201dccffe-kube-api-access-lnptb" pod "c6ba9acb-5f2f-4028-9963-fe2201dccffe" (UID: "c6ba9acb-5f2f-4028-9963-fe2201dccffe"). InnerVolumeSpecName "kube-api-access-lnptb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:26:00.072687 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:00.072616 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a55e1-7b5c564646-k975p"] Oct 30 00:26:00.121802 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:00.119655 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnptb\" (UniqueName: \"kubernetes.io/projected/c6ba9acb-5f2f-4028-9963-fe2201dccffe-kube-api-access-lnptb\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:26:00.674518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:00.674458 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-76ddd5685d-v4sqk" podUID="c6ba9acb-5f2f-4028-9963-fe2201dccffe" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.168:9004: i/o timeout" Oct 30 00:26:00.807310 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:00.807267 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-76ddd5685d-v4sqk" event={"ID":"c6ba9acb-5f2f-4028-9963-fe2201dccffe","Type":"ContainerDied","Data":"f2359c52538e174d0ff1087fff2e5ff1e2af7f3f247612a992b08c8af308d2c0"} Oct 30 00:26:00.807492 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:00.807375 5001 scope.go:122] "RemoveContainer" containerID="382793f6756a34a19463fb58d36ff50b5582b757a1fe27543ce80e7fe3c67dcb" Oct 30 00:26:00.807492 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:00.807284 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-96027e-76ddd5685d-v4sqk" Oct 30 00:26:00.858282 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:00.858244 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96027e-76ddd5685d-v4sqk"] Oct 30 00:26:00.867628 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:00.867590 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96027e-76ddd5685d-v4sqk"] Oct 30 00:26:00.920811 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:00.920779 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="badf6bc7-f9da-431f-8007-9f7b53cec3a1" path="/var/lib/kubelet/pods/badf6bc7-f9da-431f-8007-9f7b53cec3a1/volumes" Oct 30 00:26:00.921314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:00.921298 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6ba9acb-5f2f-4028-9963-fe2201dccffe" path="/var/lib/kubelet/pods/c6ba9acb-5f2f-4028-9963-fe2201dccffe/volumes" Oct 30 00:26:00.921714 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:00.921701 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8" path="/var/lib/kubelet/pods/c6d4dadc-2ba8-4e80-81c2-2c20dc3a1be8/volumes" Oct 30 00:26:00.922413 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:00.922397 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de1c480c-7dbc-47b0-bdff-d9accfbb0944" path="/var/lib/kubelet/pods/de1c480c-7dbc-47b0-bdff-d9accfbb0944/volumes" Oct 30 00:26:02.325935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.324710 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-bf5958687-wqrzq"] Oct 30 00:26:02.325935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.325818 5001 cpu_manager.go:414] "containerMap: removing container" podUID="badf6bc7-f9da-431f-8007-9f7b53cec3a1" containerName="payload-processing" Oct 30 00:26:02.325935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.325843 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="badf6bc7-f9da-431f-8007-9f7b53cec3a1" containerName="payload-processing" Oct 30 00:26:02.325935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.325860 5001 cpu_manager.go:414] "containerMap: removing container" podUID="7a4b5289-1a4b-4184-a1a7-7b11b336ae56" containerName="revoke-keys" Oct 30 00:26:02.325935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.325867 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a4b5289-1a4b-4184-a1a7-7b11b336ae56" containerName="revoke-keys" Oct 30 00:26:02.325935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.325888 5001 cpu_manager.go:414] "containerMap: removing container" podUID="c6ba9acb-5f2f-4028-9963-fe2201dccffe" containerName="payload-processing" Oct 30 00:26:02.325935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.325896 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6ba9acb-5f2f-4028-9963-fe2201dccffe" containerName="payload-processing" Oct 30 00:26:02.325935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.325926 5001 cpu_manager.go:414] "containerMap: removing container" podUID="de1c480c-7dbc-47b0-bdff-d9accfbb0944" containerName="payload-pre-processing" Oct 30 00:26:02.325935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.325934 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de1c480c-7dbc-47b0-bdff-d9accfbb0944" containerName="payload-pre-processing" Oct 30 00:26:02.342938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.325954 5001 cpu_manager.go:414] "containerMap: removing container" podUID="259e6702-4256-4bbe-b2e0-86b4d2071f66" containerName="payload-pre-processing" Oct 30 00:26:02.342938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.325962 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="259e6702-4256-4bbe-b2e0-86b4d2071f66" containerName="payload-pre-processing" Oct 30 00:26:02.342938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.325983 5001 cpu_manager.go:414] "containerMap: removing container" podUID="a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7" containerName="istio-proxy" Oct 30 00:26:02.342938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.325989 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7" containerName="istio-proxy" Oct 30 00:26:02.342938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.326015 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f91ba282-c730-4f93-ba5b-92a63bad3488" containerName="payload-pre-processing" Oct 30 00:26:02.342938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.326022 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f91ba282-c730-4f93-ba5b-92a63bad3488" containerName="payload-pre-processing" Oct 30 00:26:02.342938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.326239 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed3d03e4-5550-40bc-8c00-48c3c7927fa1" containerName="istio-proxy" Oct 30 00:26:02.342938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.326258 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f91ba282-c730-4f93-ba5b-92a63bad3488" containerName="payload-pre-processing" Oct 30 00:26:02.342938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.326269 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7" containerName="istio-proxy" Oct 30 00:26:02.342938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.326289 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="badf6bc7-f9da-431f-8007-9f7b53cec3a1" containerName="payload-processing" Oct 30 00:26:02.342938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.326298 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a4b5289-1a4b-4184-a1a7-7b11b336ae56" containerName="revoke-keys" Oct 30 00:26:02.342938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.326309 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6ba9acb-5f2f-4028-9963-fe2201dccffe" containerName="payload-processing" Oct 30 00:26:02.342938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.326320 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="de1c480c-7dbc-47b0-bdff-d9accfbb0944" containerName="payload-pre-processing" Oct 30 00:26:02.342938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.326332 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="259e6702-4256-4bbe-b2e0-86b4d2071f66" containerName="payload-pre-processing" Oct 30 00:26:02.657978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.606230 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-bf5958687-wqrzq" Oct 30 00:26:02.657978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.656873 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f8397b\"" Oct 30 00:26:02.657978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.657602 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f8397b-dockercfg-xh924\"" Oct 30 00:26:02.664137 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.658863 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmz2p\" (UniqueName: \"kubernetes.io/projected/1359a6e8-77c9-4e05-b11e-27ba0c4573fe-kube-api-access-cmz2p\") pod \"payload-pre-processing-e2e-ipp-a-f8397b-bf5958687-wqrzq\" (UID: \"1359a6e8-77c9-4e05-b11e-27ba0c4573fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-bf5958687-wqrzq" Oct 30 00:26:02.664137 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.658904 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1359a6e8-77c9-4e05-b11e-27ba0c4573fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f8397b-bf5958687-wqrzq\" (UID: \"1359a6e8-77c9-4e05-b11e-27ba0c4573fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-bf5958687-wqrzq" Oct 30 00:26:02.751235 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.751188 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk"] Oct 30 00:26:02.751764 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.751720 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" podUID="03f8426e-b14c-43f9-9598-04b5b7b309b5" containerName="istio-proxy" containerID="cri-o://cf1f6bee7420b91495207ae3fd2dabf726fc1481aaac039fc291bcf9ef71b8a4" gracePeriod=30 Oct 30 00:26:02.763457 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.761842 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmz2p\" (UniqueName: \"kubernetes.io/projected/1359a6e8-77c9-4e05-b11e-27ba0c4573fe-kube-api-access-cmz2p\") pod \"payload-pre-processing-e2e-ipp-a-f8397b-bf5958687-wqrzq\" (UID: \"1359a6e8-77c9-4e05-b11e-27ba0c4573fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-bf5958687-wqrzq" Oct 30 00:26:02.763457 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.761891 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1359a6e8-77c9-4e05-b11e-27ba0c4573fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f8397b-bf5958687-wqrzq\" (UID: \"1359a6e8-77c9-4e05-b11e-27ba0c4573fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-bf5958687-wqrzq" Oct 30 00:26:02.763741 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.763068 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1359a6e8-77c9-4e05-b11e-27ba0c4573fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f8397b-bf5958687-wqrzq\" (UID: \"1359a6e8-77c9-4e05-b11e-27ba0c4573fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-bf5958687-wqrzq" Oct 30 00:26:02.927693 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.900367 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmz2p\" (UniqueName: \"kubernetes.io/projected/1359a6e8-77c9-4e05-b11e-27ba0c4573fe-kube-api-access-cmz2p\") pod \"payload-pre-processing-e2e-ipp-a-f8397b-bf5958687-wqrzq\" (UID: \"1359a6e8-77c9-4e05-b11e-27ba0c4573fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-bf5958687-wqrzq" Oct 30 00:26:02.990024 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:02.985400 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-bf5958687-wqrzq"] Oct 30 00:26:03.048537 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:03.047774 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-bf5958687-wqrzq" Oct 30 00:26:03.711560 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:03.711330 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-bf5958687-wqrzq"] Oct 30 00:26:03.718487 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:26:03.717556 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1359a6e8_77c9_4e05_b11e_27ba0c4573fe.slice/crio-6694143f2c848ba4e9f784f3f75a4adf6431d69e3ba5e8e69701f8a4aee8d3af WatchSource:0}: Error finding container 6694143f2c848ba4e9f784f3f75a4adf6431d69e3ba5e8e69701f8a4aee8d3af: Status 404 returned error can't find the container with id 6694143f2c848ba4e9f784f3f75a4adf6431d69e3ba5e8e69701f8a4aee8d3af Oct 30 00:26:03.948010 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:03.944959 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-bf5958687-wqrzq" event={"ID":"1359a6e8-77c9-4e05-b11e-27ba0c4573fe","Type":"ContainerStarted","Data":"347f96f80d12786d86391924e2b1e64b378ff8be8741652e052737c888f119a2"} Oct 30 00:26:03.948010 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:03.945043 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-bf5958687-wqrzq" event={"ID":"1359a6e8-77c9-4e05-b11e-27ba0c4573fe","Type":"ContainerStarted","Data":"6694143f2c848ba4e9f784f3f75a4adf6431d69e3ba5e8e69701f8a4aee8d3af"} Oct 30 00:26:03.948010 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:03.945212 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-bf5958687-wqrzq" Oct 30 00:26:03.983028 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:03.982398 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-bf5958687-wqrzq" podStartSLOduration=1.982379704 podStartE2EDuration="1.982379704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:26:03.980187635 +0000 UTC m=+1279.185688815" watchObservedRunningTime="2026-10-30 00:26:03.982379704 +0000 UTC m=+1279.187880887" Oct 30 00:26:04.763937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:04.738058 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m"] Oct 30 00:26:04.763937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:04.739275 5001 cpu_manager.go:414] "containerMap: removing container" podUID="a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7" containerName="istio-proxy" Oct 30 00:26:04.763937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:04.739299 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7" containerName="istio-proxy" Oct 30 00:26:04.763937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:04.739517 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3dfd6fc-ec4f-47ae-99b4-5c1e7219bcc7" containerName="istio-proxy" Oct 30 00:26:04.904971 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:04.884108 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m"] Oct 30 00:26:04.904971 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:04.884240 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:26:05.136012 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:05.134284 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-workload-certs\") pod \"e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:26:05.136012 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:05.134393 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e34c0d4a-0def-4964-8c2a-4599941a3a03-istio-token\") pod \"e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:26:05.136012 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:05.134428 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-istio-data\") pod \"e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:26:05.136012 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:05.134547 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4w5b\" (UniqueName: \"kubernetes.io/projected/e34c0d4a-0def-4964-8c2a-4599941a3a03-kube-api-access-d4w5b\") pod \"e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:26:05.136012 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:05.134640 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-credential-socket\") pod \"e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:26:05.136012 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:05.134701 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-istio-envoy\") pod \"e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:26:05.136012 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:05.134745 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-workload-socket\") pod \"e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:26:05.136012 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:05.134858 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e34c0d4a-0def-4964-8c2a-4599941a3a03-istio-podinfo\") pod \"e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:26:05.152931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:05.152876 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-8b3bf0-openshift-default-dockercfg-6xw7x\"" Oct 30 00:26:05.188537 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:05.188504 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e34c0d4a-0def-4964-8c2a-4599941a3a03-istiod-ca-cert\") pod \"e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:26:05.310025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:05.306016 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e34c0d4a-0def-4964-8c2a-4599941a3a03-istio-token\") pod \"e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:26:05.310025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:05.306077 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-istio-data\") pod \"e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:26:05.310025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:05.306194 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4w5b\" (UniqueName: \"kubernetes.io/projected/e34c0d4a-0def-4964-8c2a-4599941a3a03-kube-api-access-d4w5b\") pod \"e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:26:05.310025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:05.306287 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-credential-socket\") pod \"e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:26:05.310025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:05.306329 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-istio-envoy\") pod \"e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:26:05.310025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:05.306373 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-workload-socket\") pod \"e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:26:05.310025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:05.306469 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e34c0d4a-0def-4964-8c2a-4599941a3a03-istio-podinfo\") pod \"e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:26:05.310025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:05.306507 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e34c0d4a-0def-4964-8c2a-4599941a3a03-istiod-ca-cert\") pod \"e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:26:05.310025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:05.306583 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-workload-certs\") pod \"e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:26:05.312217 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:05.310984 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-workload-certs\") pod \"e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:26:05.312217 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:05.311500 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-istio-data\") pod \"e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:26:05.312217 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:05.311871 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-credential-socket\") pod \"e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:26:05.317015 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:05.316985 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-workload-socket\") pod \"e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:26:05.317442 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:05.317422 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e34c0d4a-0def-4964-8c2a-4599941a3a03-istiod-ca-cert\") pod \"e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:26:05.360948 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:05.351687 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-istio-envoy\") pod \"e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:26:05.364932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:05.362225 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e34c0d4a-0def-4964-8c2a-4599941a3a03-istio-podinfo\") pod \"e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:26:05.436004 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:05.427934 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4w5b\" (UniqueName: \"kubernetes.io/projected/e34c0d4a-0def-4964-8c2a-4599941a3a03-kube-api-access-d4w5b\") pod \"e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:26:05.436004 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:05.428525 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e34c0d4a-0def-4964-8c2a-4599941a3a03-istio-token\") pod \"e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:26:05.798987 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:05.781140 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:26:05.798987 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:05.781209 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:26:05.968113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:05.968082 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:26:06.068537 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.057165 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4"] Oct 30 00:26:06.068537 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.068324 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:26:06.088243 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.087319 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-6cfcf7-openshift-default-dockercfg-vxfhf\"" Oct 30 00:26:06.098120 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.092490 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4"] Oct 30 00:26:06.123149 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.123076 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88f86bc7-z48bk"] Oct 30 00:26:06.134452 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.134416 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88f86bc7-z48bk"] Oct 30 00:26:06.168448 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.168416 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-workload-socket\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:26:06.169063 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.169041 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8vrn\" (UniqueName: \"kubernetes.io/projected/0c35f644-80ac-439f-9efd-631b26b20a0c-kube-api-access-d8vrn\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:26:06.169252 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.169225 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-workload-certs\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:26:06.169516 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.169483 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-credential-socket\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:26:06.186319 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.186277 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c35f644-80ac-439f-9efd-631b26b20a0c-istio-token\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:26:06.186610 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.186573 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c35f644-80ac-439f-9efd-631b26b20a0c-istiod-ca-cert\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:26:06.186762 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.186746 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-istio-envoy\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:26:06.186897 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.186881 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c35f644-80ac-439f-9efd-631b26b20a0c-istio-podinfo\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:26:06.187321 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.187298 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-istio-data\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:26:06.188063 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.188042 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bef0d341-znp5p"] Oct 30 00:26:06.388837 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.387499 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-istio-data\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:26:06.388837 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.387715 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-workload-socket\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:26:06.388837 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.387776 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8vrn\" (UniqueName: \"kubernetes.io/projected/0c35f644-80ac-439f-9efd-631b26b20a0c-kube-api-access-d8vrn\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:26:06.388837 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.387820 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-workload-certs\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:26:06.388837 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.387949 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-credential-socket\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:26:06.388837 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.388019 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c35f644-80ac-439f-9efd-631b26b20a0c-istio-token\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:26:06.388837 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.388092 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c35f644-80ac-439f-9efd-631b26b20a0c-istiod-ca-cert\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:26:06.388837 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.388124 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-istio-envoy\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:26:06.388837 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.388165 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c35f644-80ac-439f-9efd-631b26b20a0c-istio-podinfo\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:26:06.393631 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.389511 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-workload-certs\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:26:06.393631 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.389782 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-istio-data\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:26:06.393631 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.390061 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-workload-socket\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:26:06.393631 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.390940 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-credential-socket\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:26:06.393631 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.392149 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c35f644-80ac-439f-9efd-631b26b20a0c-istiod-ca-cert\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:26:06.403975 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.403098 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bef0d341-znp5p"] Oct 30 00:26:06.421254 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.421199 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8vrn\" (UniqueName: \"kubernetes.io/projected/0c35f644-80ac-439f-9efd-631b26b20a0c-kube-api-access-d8vrn\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:26:06.424886 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.424859 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c35f644-80ac-439f-9efd-631b26b20a0c-istio-token\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:26:06.437893 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.430575 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c35f644-80ac-439f-9efd-631b26b20a0c-istio-podinfo\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:26:06.549772 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.549677 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-istio-envoy\") pod \"e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:26:06.716540 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.716227 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:26:06.942240 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.939824 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bec19ed-2ef0-41d3-a53c-6275eea40fb9" path="/var/lib/kubelet/pods/2bec19ed-2ef0-41d3-a53c-6275eea40fb9/volumes" Oct 30 00:26:06.942240 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:06.940616 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eadd5f6-4577-4b82-8903-24648a5493e6" path="/var/lib/kubelet/pods/5eadd5f6-4577-4b82-8903-24648a5493e6/volumes" Oct 30 00:26:07.148711 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:07.148676 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m"] Oct 30 00:26:07.186273 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:07.171988 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:26:07.186273 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:07.172084 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:26:07.186273 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:07.172126 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:26:07.517984 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:07.517840 5001 generic.go:356] "Generic (PLEG): container finished" podID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerID="7735f81688c824ce4c03de2ab668e577ef7e34cae26d89e81364b4b573038fcc" exitCode=0 Oct 30 00:26:07.520251 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:07.520218 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" event={"ID":"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb","Type":"ContainerDied","Data":"7735f81688c824ce4c03de2ab668e577ef7e34cae26d89e81364b4b573038fcc"} Oct 30 00:26:07.520444 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:07.520425 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" event={"ID":"e34c0d4a-0def-4964-8c2a-4599941a3a03","Type":"ContainerStarted","Data":"94d87f46b63530846c1ec019e04a61561fe762b6d21c4605ea64af8a4b36a10b"} Oct 30 00:26:07.521074 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:07.521057 5001 scope.go:122] "RemoveContainer" containerID="7735f81688c824ce4c03de2ab668e577ef7e34cae26d89e81364b4b573038fcc" Oct 30 00:26:07.524683 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:07.524645 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:26:07.525227 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:07.525209 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:26:07.525355 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:07.525341 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:26:07.627992 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:26:07.625202 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c35f644_80ac_439f_9efd_631b26b20a0c.slice/crio-b33358192501ebcb0160abf97d1494f6dc0e0b1aebb00f82881335dbd9a6a047 WatchSource:0}: Error finding container b33358192501ebcb0160abf97d1494f6dc0e0b1aebb00f82881335dbd9a6a047: Status 404 returned error can't find the container with id b33358192501ebcb0160abf97d1494f6dc0e0b1aebb00f82881335dbd9a6a047 Oct 30 00:26:07.640412 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:07.640360 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:26:07.640562 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:07.640461 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:26:07.640562 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:07.640504 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:26:07.651058 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:07.649019 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4"] Oct 30 00:26:07.654553 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:07.654519 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg"] Oct 30 00:26:07.654738 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:07.654718 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" podUID="3868e018-e83d-4eb5-9aff-842760b30295" containerName="istio-proxy" containerID="cri-o://e32e20f75276058c14000dcb4d807d66a65154813c35ecf95df0794e7429c10e" gracePeriod=30 Oct 30 00:26:08.201265 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.201071 5001 patch_prober.go:29] interesting pod/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 00:26:08.201265 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.201119 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" podUID="03f8426e-b14c-43f9-9598-04b5b7b309b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 00:26:08.538423 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.536691 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" event={"ID":"e34c0d4a-0def-4964-8c2a-4599941a3a03","Type":"ContainerStarted","Data":"6171c8650a197691769b8e0b1b0674f8651df0f15863762f9b90d8d9ec49c156"} Oct 30 00:26:08.539954 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.539896 5001 generic.go:356] "Generic (PLEG): container finished" podID="03f8426e-b14c-43f9-9598-04b5b7b309b5" containerID="cf1f6bee7420b91495207ae3fd2dabf726fc1481aaac039fc291bcf9ef71b8a4" exitCode=0 Oct 30 00:26:08.540243 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.540122 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" event={"ID":"03f8426e-b14c-43f9-9598-04b5b7b309b5","Type":"ContainerDied","Data":"cf1f6bee7420b91495207ae3fd2dabf726fc1481aaac039fc291bcf9ef71b8a4"} Oct 30 00:26:08.540355 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.540289 5001 scope.go:122] "RemoveContainer" containerID="13b59302f73b01109636fec53bf8d139d4c2af5430bf221ebba254a665d0294c" Oct 30 00:26:08.543317 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.543289 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" event={"ID":"0c35f644-80ac-439f-9efd-631b26b20a0c","Type":"ContainerStarted","Data":"eee8c4d26b92ea429508ab9169296ac584476f24181e6e3c79ae36f85e5a2b3d"} Oct 30 00:26:08.543519 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.543500 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" event={"ID":"0c35f644-80ac-439f-9efd-631b26b20a0c","Type":"ContainerStarted","Data":"b33358192501ebcb0160abf97d1494f6dc0e0b1aebb00f82881335dbd9a6a047"} Oct 30 00:26:08.546703 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.546677 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" event={"ID":"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb","Type":"ContainerStarted","Data":"0dfbf990a78ac40811b9bb64141e12639acd028f98846a2c82fd321d74994aa1"} Oct 30 00:26:08.586763 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.586716 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" podStartSLOduration=4.586699894 podStartE2EDuration="4.586699894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:26:08.578052147 +0000 UTC m=+1283.783553329" watchObservedRunningTime="2026-10-30 00:26:08.586699894 +0000 UTC m=+1283.792201075" Oct 30 00:26:08.676066 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.676002 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" podStartSLOduration=3.675979961 podStartE2EDuration="3.675979961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:26:08.667924388 +0000 UTC m=+1283.873425564" watchObservedRunningTime="2026-10-30 00:26:08.675979961 +0000 UTC m=+1283.881481143" Oct 30 00:26:08.715189 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.715155 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:26:08.717406 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.717377 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:26:08.719166 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.718848 5001 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 00:26:08.719166 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.718900 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 00:26:08.732284 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.732257 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03f8426e-b14c-43f9-9598-04b5b7b309b5-istio-token\" (UniqueName: \"kubernetes.io/projected/03f8426e-b14c-43f9-9598-04b5b7b309b5-istio-token\") pod \"03f8426e-b14c-43f9-9598-04b5b7b309b5\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " Oct 30 00:26:08.732421 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.732367 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-credential-socket\") pod \"03f8426e-b14c-43f9-9598-04b5b7b309b5\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " Oct 30 00:26:08.732421 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.732416 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-workload-socket\") pod \"03f8426e-b14c-43f9-9598-04b5b7b309b5\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " Oct 30 00:26:08.732560 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.732441 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03f8426e-b14c-43f9-9598-04b5b7b309b5-kube-api-access-9jp7x\" (UniqueName: \"kubernetes.io/projected/03f8426e-b14c-43f9-9598-04b5b7b309b5-kube-api-access-9jp7x\") pod \"03f8426e-b14c-43f9-9598-04b5b7b309b5\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " Oct 30 00:26:08.732619 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.732567 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-istio-envoy\") pod \"03f8426e-b14c-43f9-9598-04b5b7b309b5\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " Oct 30 00:26:08.732677 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.732637 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-workload-certs\") pod \"03f8426e-b14c-43f9-9598-04b5b7b309b5\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " Oct 30 00:26:08.732732 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.732696 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03f8426e-b14c-43f9-9598-04b5b7b309b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f8426e-b14c-43f9-9598-04b5b7b309b5-istiod-ca-cert\") pod \"03f8426e-b14c-43f9-9598-04b5b7b309b5\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " Oct 30 00:26:08.732788 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.732753 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-credential-socket" pod "03f8426e-b14c-43f9-9598-04b5b7b309b5" (UID: "03f8426e-b14c-43f9-9598-04b5b7b309b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:26:08.732788 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.732768 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-workload-socket" pod "03f8426e-b14c-43f9-9598-04b5b7b309b5" (UID: "03f8426e-b14c-43f9-9598-04b5b7b309b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:26:08.732902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.732775 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-istio-data\") pod \"03f8426e-b14c-43f9-9598-04b5b7b309b5\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " Oct 30 00:26:08.732902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.732856 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03f8426e-b14c-43f9-9598-04b5b7b309b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f8426e-b14c-43f9-9598-04b5b7b309b5-istio-podinfo\") pod \"03f8426e-b14c-43f9-9598-04b5b7b309b5\" (UID: \"03f8426e-b14c-43f9-9598-04b5b7b309b5\") " Oct 30 00:26:08.733036 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.732981 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-istio-data" pod "03f8426e-b14c-43f9-9598-04b5b7b309b5" (UID: "03f8426e-b14c-43f9-9598-04b5b7b309b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:26:08.733207 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.733184 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-workload-certs" pod "03f8426e-b14c-43f9-9598-04b5b7b309b5" (UID: "03f8426e-b14c-43f9-9598-04b5b7b309b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:26:08.733599 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.733488 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03f8426e-b14c-43f9-9598-04b5b7b309b5-istiod-ca-cert" pod "03f8426e-b14c-43f9-9598-04b5b7b309b5" (UID: "03f8426e-b14c-43f9-9598-04b5b7b309b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:26:08.734198 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.734179 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:26:08.734305 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.734201 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:26:08.734305 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.734215 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:26:08.734305 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.734228 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f8426e-b14c-43f9-9598-04b5b7b309b5-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:26:08.734305 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.734240 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:26:08.735312 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.735283 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f8426e-b14c-43f9-9598-04b5b7b309b5-istio-token" pod "03f8426e-b14c-43f9-9598-04b5b7b309b5" (UID: "03f8426e-b14c-43f9-9598-04b5b7b309b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:26:08.735430 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.735353 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-istio-envoy" pod "03f8426e-b14c-43f9-9598-04b5b7b309b5" (UID: "03f8426e-b14c-43f9-9598-04b5b7b309b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:26:08.735780 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.735756 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f8426e-b14c-43f9-9598-04b5b7b309b5-kube-api-access-9jp7x" pod "03f8426e-b14c-43f9-9598-04b5b7b309b5" (UID: "03f8426e-b14c-43f9-9598-04b5b7b309b5"). InnerVolumeSpecName "kube-api-access-9jp7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:26:08.735936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.735853 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03f8426e-b14c-43f9-9598-04b5b7b309b5-istio-podinfo" pod "03f8426e-b14c-43f9-9598-04b5b7b309b5" (UID: "03f8426e-b14c-43f9-9598-04b5b7b309b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:26:08.742762 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.742738 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:26:08.743812 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.743792 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 00:26:08.743932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.743834 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 00:26:08.893547 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.893442 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f8426e-b14c-43f9-9598-04b5b7b309b5-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:26:08.893547 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.893481 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03f8426e-b14c-43f9-9598-04b5b7b309b5-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:26:08.893547 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.893499 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jp7x\" (UniqueName: \"kubernetes.io/projected/03f8426e-b14c-43f9-9598-04b5b7b309b5-kube-api-access-9jp7x\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:26:08.893547 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.893513 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f8426e-b14c-43f9-9598-04b5b7b309b5-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:26:08.996324 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.996273 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:26:08.998284 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.998255 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 00:26:08.998547 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:08.998491 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 00:26:09.559237 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.559210 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" Oct 30 00:26:09.559729 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.559242 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk" event={"ID":"03f8426e-b14c-43f9-9598-04b5b7b309b5","Type":"ContainerDied","Data":"4f195e056c50eee7bb811b0c5fdc2aec666594d56c673b5556ee7a4c9bed2bf4"} Oct 30 00:26:09.559729 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.559299 5001 scope.go:122] "RemoveContainer" containerID="cf1f6bee7420b91495207ae3fd2dabf726fc1481aaac039fc291bcf9ef71b8a4" Oct 30 00:26:09.581575 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.581549 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:26:09.581765 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.581747 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" podUID="3868e018-e83d-4eb5-9aff-842760b30295" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:26:09.624443 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.624405 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk"] Oct 30 00:26:09.637005 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.636970 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-08dd33-openshift-default-6f49cd5cf-996pk"] Oct 30 00:26:09.717159 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.717098 5001 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 00:26:09.717332 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.717182 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 00:26:09.721537 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.721492 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8"] Oct 30 00:26:09.722585 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.722558 5001 cpu_manager.go:414] "containerMap: removing container" podUID="03f8426e-b14c-43f9-9598-04b5b7b309b5" containerName="istio-proxy" Oct 30 00:26:09.722730 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.722583 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03f8426e-b14c-43f9-9598-04b5b7b309b5" containerName="istio-proxy" Oct 30 00:26:09.722843 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.722829 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="03f8426e-b14c-43f9-9598-04b5b7b309b5" containerName="istio-proxy" Oct 30 00:26:09.728777 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.728756 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:26:09.743315 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.743286 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 00:26:09.743466 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.743339 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 00:26:09.748842 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.748797 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-522ea7-openshift-default-dockercfg-zf9rx\"" Oct 30 00:26:09.756481 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.756441 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8"] Oct 30 00:26:09.817874 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.817789 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-istio-data\") pod \"e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:26:09.817874 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.817826 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-workload-socket\") pod \"e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:26:09.817874 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.817849 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-credential-socket\") pod \"e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:26:09.818157 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.817938 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nks6s\" (UniqueName: \"kubernetes.io/projected/4fb15e40-3233-4329-917c-6bfcf33cb894-kube-api-access-nks6s\") pod \"e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:26:09.818157 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.818081 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-workload-certs\") pod \"e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:26:09.818157 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.818113 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-istio-envoy\") pod \"e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:26:09.818260 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.818237 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fb15e40-3233-4329-917c-6bfcf33cb894-istio-podinfo\") pod \"e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:26:09.818416 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.818400 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fb15e40-3233-4329-917c-6bfcf33cb894-istio-token\") pod \"e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:26:09.818453 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.818424 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fb15e40-3233-4329-917c-6bfcf33cb894-istiod-ca-cert\") pod \"e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:26:09.925999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.922305 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fb15e40-3233-4329-917c-6bfcf33cb894-istio-podinfo\") pod \"e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:26:09.925999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.922535 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fb15e40-3233-4329-917c-6bfcf33cb894-istio-token\") pod \"e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:26:09.925999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.922569 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fb15e40-3233-4329-917c-6bfcf33cb894-istiod-ca-cert\") pod \"e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:26:09.925999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.922643 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-istio-data\") pod \"e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:26:09.925999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.922670 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-workload-socket\") pod \"e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:26:09.925999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.922696 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-credential-socket\") pod \"e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:26:09.925999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.922720 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nks6s\" (UniqueName: \"kubernetes.io/projected/4fb15e40-3233-4329-917c-6bfcf33cb894-kube-api-access-nks6s\") pod \"e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:26:09.925999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.922857 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-workload-certs\") pod \"e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:26:09.925999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.922888 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-istio-envoy\") pod \"e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:26:09.940006 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.927313 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-istio-data\") pod \"e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:26:09.940006 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.933275 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-credential-socket\") pod \"e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:26:09.940006 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.933536 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-workload-socket\") pod \"e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:26:09.940006 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.934097 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-workload-certs\") pod \"e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:26:09.948937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.940566 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fb15e40-3233-4329-917c-6bfcf33cb894-istiod-ca-cert\") pod \"e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:26:09.980950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.974088 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fb15e40-3233-4329-917c-6bfcf33cb894-istio-podinfo\") pod \"e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:26:09.980950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.974466 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-istio-envoy\") pod \"e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:26:09.985943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.984903 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nks6s\" (UniqueName: \"kubernetes.io/projected/4fb15e40-3233-4329-917c-6bfcf33cb894-kube-api-access-nks6s\") pod \"e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:26:09.987996 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.987964 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fb15e40-3233-4329-917c-6bfcf33cb894-istio-token\") pod \"e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:26:10.024935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.999748 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 00:26:10.024935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:09.999809 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 00:26:10.072394 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:10.072305 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:26:10.101674 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:10.101571 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59167bf4-fkjd2"] Oct 30 00:26:10.233236 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:10.186254 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59167bf4-fkjd2"] Oct 30 00:26:10.737147 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:10.736009 5001 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 00:26:10.737147 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:10.736068 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 00:26:10.747343 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:10.746248 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 00:26:10.747343 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:10.746303 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 00:26:10.909469 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:26:10.909436 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fb15e40_3233_4329_917c_6bfcf33cb894.slice/crio-a4b639d6487d66bb6bfbbf4c0c743bbababef912ed1a60e33d5d9d323d4b15d4 WatchSource:0}: Error finding container a4b639d6487d66bb6bfbbf4c0c743bbababef912ed1a60e33d5d9d323d4b15d4: Status 404 returned error can't find the container with id a4b639d6487d66bb6bfbbf4c0c743bbababef912ed1a60e33d5d9d323d4b15d4 Oct 30 00:26:10.913824 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:10.913782 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:26:10.913963 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:10.913876 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:26:10.914038 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:10.913938 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:26:10.924272 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:10.924236 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03f8426e-b14c-43f9-9598-04b5b7b309b5" path="/var/lib/kubelet/pods/03f8426e-b14c-43f9-9598-04b5b7b309b5/volumes" Oct 30 00:26:10.925061 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:10.925037 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5a2843b-681e-4663-a1ea-020c50c1d701" path="/var/lib/kubelet/pods/d5a2843b-681e-4663-a1ea-020c50c1d701/volumes" Oct 30 00:26:10.925559 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:10.925538 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8"] Oct 30 00:26:10.995304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:10.995272 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 00:26:10.995465 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:10.995324 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 00:26:11.628002 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:11.627960 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" event={"ID":"4fb15e40-3233-4329-917c-6bfcf33cb894","Type":"ContainerStarted","Data":"72ce3749930f3cca3cbe89433546c1d3a3949e2181da0067d680dad946f357b7"} Oct 30 00:26:11.628175 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:11.628021 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" event={"ID":"4fb15e40-3233-4329-917c-6bfcf33cb894","Type":"ContainerStarted","Data":"a4b639d6487d66bb6bfbbf4c0c743bbababef912ed1a60e33d5d9d323d4b15d4"} Oct 30 00:26:11.691195 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:11.691142 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" podStartSLOduration=2.691127455 podStartE2EDuration="2.691127455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:26:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:26:11.676437308 +0000 UTC m=+1286.881938506" watchObservedRunningTime="2026-10-30 00:26:11.691127455 +0000 UTC m=+1286.896628635" Oct 30 00:26:11.717961 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:11.717921 5001 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 00:26:11.718126 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:11.717983 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 00:26:11.742411 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:11.742374 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 00:26:11.742941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:11.742437 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 00:26:11.995789 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:11.995709 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 00:26:11.995789 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:11.995764 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 00:26:12.074944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:12.074871 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:26:12.081547 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:12.081515 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:26:12.638301 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:12.638267 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:26:12.639383 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:12.639359 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:26:12.726035 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:12.725495 5001 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 00:26:12.726035 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:12.725557 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 00:26:12.742585 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:12.742455 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 00:26:12.742585 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:12.742517 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 00:26:12.990693 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:12.990659 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:26:12.995502 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:12.995471 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 00:26:12.995618 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:12.995525 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 00:26:13.082881 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.082828 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3868e018-e83d-4eb5-9aff-842760b30295-istio-token\" (UniqueName: \"kubernetes.io/projected/3868e018-e83d-4eb5-9aff-842760b30295-istio-token\") pod \"3868e018-e83d-4eb5-9aff-842760b30295\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " Oct 30 00:26:13.083090 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.082971 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-workload-certs\") pod \"3868e018-e83d-4eb5-9aff-842760b30295\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " Oct 30 00:26:13.083090 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.083015 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3868e018-e83d-4eb5-9aff-842760b30295-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3868e018-e83d-4eb5-9aff-842760b30295-istio-podinfo\") pod \"3868e018-e83d-4eb5-9aff-842760b30295\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " Oct 30 00:26:13.083090 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.083047 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3868e018-e83d-4eb5-9aff-842760b30295-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3868e018-e83d-4eb5-9aff-842760b30295-istiod-ca-cert\") pod \"3868e018-e83d-4eb5-9aff-842760b30295\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " Oct 30 00:26:13.083270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.083105 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-workload-socket\") pod \"3868e018-e83d-4eb5-9aff-842760b30295\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " Oct 30 00:26:13.083333 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.083267 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-credential-socket\") pod \"3868e018-e83d-4eb5-9aff-842760b30295\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " Oct 30 00:26:13.083391 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.083329 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-istio-data\") pod \"3868e018-e83d-4eb5-9aff-842760b30295\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " Oct 30 00:26:13.083447 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.083428 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-istio-envoy\") pod \"3868e018-e83d-4eb5-9aff-842760b30295\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " Oct 30 00:26:13.083550 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.083527 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3868e018-e83d-4eb5-9aff-842760b30295-kube-api-access-mnkn8\" (UniqueName: \"kubernetes.io/projected/3868e018-e83d-4eb5-9aff-842760b30295-kube-api-access-mnkn8\") pod \"3868e018-e83d-4eb5-9aff-842760b30295\" (UID: \"3868e018-e83d-4eb5-9aff-842760b30295\") " Oct 30 00:26:13.083974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.083941 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3868e018-e83d-4eb5-9aff-842760b30295-istiod-ca-cert" pod "3868e018-e83d-4eb5-9aff-842760b30295" (UID: "3868e018-e83d-4eb5-9aff-842760b30295"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:26:13.084246 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.084220 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-workload-certs" pod "3868e018-e83d-4eb5-9aff-842760b30295" (UID: "3868e018-e83d-4eb5-9aff-842760b30295"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:26:13.084329 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.084260 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-credential-socket" pod "3868e018-e83d-4eb5-9aff-842760b30295" (UID: "3868e018-e83d-4eb5-9aff-842760b30295"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:26:13.084511 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.084350 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:26:13.084511 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.084443 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-istio-data" pod "3868e018-e83d-4eb5-9aff-842760b30295" (UID: "3868e018-e83d-4eb5-9aff-842760b30295"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:26:13.084511 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.084457 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-workload-socket" pod "3868e018-e83d-4eb5-9aff-842760b30295" (UID: "3868e018-e83d-4eb5-9aff-842760b30295"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:26:13.087695 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.084369 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3868e018-e83d-4eb5-9aff-842760b30295-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:26:13.090396 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.090336 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-istio-envoy" pod "3868e018-e83d-4eb5-9aff-842760b30295" (UID: "3868e018-e83d-4eb5-9aff-842760b30295"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:26:13.091876 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.091837 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3868e018-e83d-4eb5-9aff-842760b30295-kube-api-access-mnkn8" pod "3868e018-e83d-4eb5-9aff-842760b30295" (UID: "3868e018-e83d-4eb5-9aff-842760b30295"). InnerVolumeSpecName "kube-api-access-mnkn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:26:13.093627 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.093453 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3868e018-e83d-4eb5-9aff-842760b30295-istio-podinfo" pod "3868e018-e83d-4eb5-9aff-842760b30295" (UID: "3868e018-e83d-4eb5-9aff-842760b30295"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:26:13.095119 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.095093 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3868e018-e83d-4eb5-9aff-842760b30295-istio-token" pod "3868e018-e83d-4eb5-9aff-842760b30295" (UID: "3868e018-e83d-4eb5-9aff-842760b30295"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:26:13.189891 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.189845 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnkn8\" (UniqueName: \"kubernetes.io/projected/3868e018-e83d-4eb5-9aff-842760b30295-kube-api-access-mnkn8\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:26:13.189891 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.189881 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3868e018-e83d-4eb5-9aff-842760b30295-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:26:13.189891 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.189891 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3868e018-e83d-4eb5-9aff-842760b30295-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:26:13.189891 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.189900 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:26:13.190170 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.189920 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:26:13.190170 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.189929 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:26:13.190170 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.189937 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3868e018-e83d-4eb5-9aff-842760b30295-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:26:13.651058 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.651024 5001 generic.go:356] "Generic (PLEG): container finished" podID="3868e018-e83d-4eb5-9aff-842760b30295" containerID="e32e20f75276058c14000dcb4d807d66a65154813c35ecf95df0794e7429c10e" exitCode=0 Oct 30 00:26:13.651342 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.651095 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" Oct 30 00:26:13.651342 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.651107 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" event={"ID":"3868e018-e83d-4eb5-9aff-842760b30295","Type":"ContainerDied","Data":"e32e20f75276058c14000dcb4d807d66a65154813c35ecf95df0794e7429c10e"} Oct 30 00:26:13.651342 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.651199 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg" event={"ID":"3868e018-e83d-4eb5-9aff-842760b30295","Type":"ContainerDied","Data":"e4e4936facaa988859d5ca88458f9f24c7d19712b9f3421f6e6da385dfb85a08"} Oct 30 00:26:13.651342 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.651256 5001 scope.go:122] "RemoveContainer" containerID="e32e20f75276058c14000dcb4d807d66a65154813c35ecf95df0794e7429c10e" Oct 30 00:26:13.670033 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.670010 5001 scope.go:122] "RemoveContainer" containerID="a8e31ad8114053cd4b4a790aa3a7450c47ae87188ac54c5dce8602bab07e312b" Oct 30 00:26:13.696136 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.685769 5001 scope.go:122] "RemoveContainer" containerID="e32e20f75276058c14000dcb4d807d66a65154813c35ecf95df0794e7429c10e" Oct 30 00:26:13.696136 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:26:13.686217 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e32e20f75276058c14000dcb4d807d66a65154813c35ecf95df0794e7429c10e\": container with ID starting with e32e20f75276058c14000dcb4d807d66a65154813c35ecf95df0794e7429c10e not found: ID does not exist" containerID="e32e20f75276058c14000dcb4d807d66a65154813c35ecf95df0794e7429c10e" Oct 30 00:26:13.696136 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.686251 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e32e20f75276058c14000dcb4d807d66a65154813c35ecf95df0794e7429c10e"} err="failed to get container status \"e32e20f75276058c14000dcb4d807d66a65154813c35ecf95df0794e7429c10e\": rpc error: code = NotFound desc = could not find container \"e32e20f75276058c14000dcb4d807d66a65154813c35ecf95df0794e7429c10e\": container with ID starting with e32e20f75276058c14000dcb4d807d66a65154813c35ecf95df0794e7429c10e not found: ID does not exist" Oct 30 00:26:13.696136 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.686275 5001 scope.go:122] "RemoveContainer" containerID="a8e31ad8114053cd4b4a790aa3a7450c47ae87188ac54c5dce8602bab07e312b" Oct 30 00:26:13.696136 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:26:13.686572 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8e31ad8114053cd4b4a790aa3a7450c47ae87188ac54c5dce8602bab07e312b\": container with ID starting with a8e31ad8114053cd4b4a790aa3a7450c47ae87188ac54c5dce8602bab07e312b not found: ID does not exist" containerID="a8e31ad8114053cd4b4a790aa3a7450c47ae87188ac54c5dce8602bab07e312b" Oct 30 00:26:13.696136 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.686598 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8e31ad8114053cd4b4a790aa3a7450c47ae87188ac54c5dce8602bab07e312b"} err="failed to get container status \"a8e31ad8114053cd4b4a790aa3a7450c47ae87188ac54c5dce8602bab07e312b\": rpc error: code = NotFound desc = could not find container \"a8e31ad8114053cd4b4a790aa3a7450c47ae87188ac54c5dce8602bab07e312b\": container with ID starting with a8e31ad8114053cd4b4a790aa3a7450c47ae87188ac54c5dce8602bab07e312b not found: ID does not exist" Oct 30 00:26:13.707026 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.706996 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg"] Oct 30 00:26:13.716422 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.716390 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-96027e-openshift-default-76ffc4cc67-55qqg"] Oct 30 00:26:13.717653 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.717626 5001 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 00:26:13.717760 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.717694 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 00:26:13.743187 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.743159 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 00:26:13.743533 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.743208 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 00:26:14.034998 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:13.999957 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 00:26:14.034998 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:14.000028 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 00:26:14.717390 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:14.717360 5001 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 00:26:14.717585 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:14.717424 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 00:26:14.742877 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:14.742841 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 00:26:14.743065 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:14.742923 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 00:26:14.919323 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:14.919287 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3868e018-e83d-4eb5-9aff-842760b30295" path="/var/lib/kubelet/pods/3868e018-e83d-4eb5-9aff-842760b30295/volumes" Oct 30 00:26:14.995270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:14.995181 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 00:26:14.995270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:14.995247 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 00:26:15.405646 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:15.405613 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-bf5958687-wqrzq" Oct 30 00:26:15.719147 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:15.719050 5001 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 00:26:15.719389 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:15.719357 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 00:26:15.768940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:15.766124 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 00:26:15.768940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:15.766225 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 00:26:15.995776 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:15.995690 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 00:26:15.995776 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:15.995752 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 00:26:16.717979 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:16.717931 5001 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 00:26:16.718152 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:16.717996 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 00:26:16.742450 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:16.742419 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 00:26:16.742615 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:16.742466 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 00:26:17.056018 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:17.053927 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 00:26:17.056018 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:17.054007 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 00:26:17.371221 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:26:17.371128 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:26:17.688797 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:17.688720 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-522ea7-7dfd54bbf5-2frb5"] Oct 30 00:26:17.689463 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:17.689447 5001 cpu_manager.go:414] "containerMap: removing container" podUID="3868e018-e83d-4eb5-9aff-842760b30295" containerName="istio-proxy" Oct 30 00:26:17.689463 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:17.689463 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3868e018-e83d-4eb5-9aff-842760b30295" containerName="istio-proxy" Oct 30 00:26:17.689561 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:17.689478 5001 cpu_manager.go:414] "containerMap: removing container" podUID="3868e018-e83d-4eb5-9aff-842760b30295" containerName="istio-proxy" Oct 30 00:26:17.689561 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:17.689485 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3868e018-e83d-4eb5-9aff-842760b30295" containerName="istio-proxy" Oct 30 00:26:17.689561 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:17.689494 5001 cpu_manager.go:414] "containerMap: removing container" podUID="03f8426e-b14c-43f9-9598-04b5b7b309b5" containerName="istio-proxy" Oct 30 00:26:17.689561 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:17.689499 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03f8426e-b14c-43f9-9598-04b5b7b309b5" containerName="istio-proxy" Oct 30 00:26:17.689690 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:17.689612 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="03f8426e-b14c-43f9-9598-04b5b7b309b5" containerName="istio-proxy" Oct 30 00:26:17.689690 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:17.689622 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3868e018-e83d-4eb5-9aff-842760b30295" containerName="istio-proxy" Oct 30 00:26:17.689690 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:17.689630 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3868e018-e83d-4eb5-9aff-842760b30295" containerName="istio-proxy" Oct 30 00:26:17.695071 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:17.695048 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-522ea7-7dfd54bbf5-2frb5" Oct 30 00:26:17.708328 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:17.708298 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-522ea7\"" Oct 30 00:26:17.717508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:17.717481 5001 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 00:26:17.717640 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:17.717528 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 00:26:17.718299 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:17.718270 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-522ea7-dockercfg-hlf5w\"" Oct 30 00:26:17.724573 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:17.724548 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-522ea7-7dfd54bbf5-2frb5"] Oct 30 00:26:17.742923 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:17.742875 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 00:26:17.743180 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:17.743151 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 00:26:17.773525 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:17.773483 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/794f4878-3fd8-4f9c-8167-98b6990270cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-522ea7-7dfd54bbf5-2frb5\" (UID: \"794f4878-3fd8-4f9c-8167-98b6990270cf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-522ea7-7dfd54bbf5-2frb5" Oct 30 00:26:17.793950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:17.774087 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv46s\" (UniqueName: \"kubernetes.io/projected/794f4878-3fd8-4f9c-8167-98b6990270cf-kube-api-access-kv46s\") pod \"payload-pre-processing-e2e-praxis-a-522ea7-7dfd54bbf5-2frb5\" (UID: \"794f4878-3fd8-4f9c-8167-98b6990270cf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-522ea7-7dfd54bbf5-2frb5" Oct 30 00:26:17.997384 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:17.997292 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 00:26:17.997641 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:17.997612 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 00:26:18.034939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:17.998480 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv46s\" (UniqueName: \"kubernetes.io/projected/794f4878-3fd8-4f9c-8167-98b6990270cf-kube-api-access-kv46s\") pod \"payload-pre-processing-e2e-praxis-a-522ea7-7dfd54bbf5-2frb5\" (UID: \"794f4878-3fd8-4f9c-8167-98b6990270cf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-522ea7-7dfd54bbf5-2frb5" Oct 30 00:26:18.042939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.042126 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/794f4878-3fd8-4f9c-8167-98b6990270cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-522ea7-7dfd54bbf5-2frb5\" (UID: \"794f4878-3fd8-4f9c-8167-98b6990270cf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-522ea7-7dfd54bbf5-2frb5" Oct 30 00:26:18.043237 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.043150 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/794f4878-3fd8-4f9c-8167-98b6990270cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-522ea7-7dfd54bbf5-2frb5\" (UID: \"794f4878-3fd8-4f9c-8167-98b6990270cf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-522ea7-7dfd54bbf5-2frb5" Oct 30 00:26:18.093863 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.093820 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv46s\" (UniqueName: \"kubernetes.io/projected/794f4878-3fd8-4f9c-8167-98b6990270cf-kube-api-access-kv46s\") pod \"payload-pre-processing-e2e-praxis-a-522ea7-7dfd54bbf5-2frb5\" (UID: \"794f4878-3fd8-4f9c-8167-98b6990270cf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-522ea7-7dfd54bbf5-2frb5" Oct 30 00:26:18.128933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.096688 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-522ea7-7dfd54bbf5-2frb5" Oct 30 00:26:18.211891 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.211741 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-522ea7-7dcbbd964-4x4lf"] Oct 30 00:26:18.218771 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.218742 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-522ea7-7dcbbd964-4x4lf" Oct 30 00:26:18.251265 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.251194 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-522ea7-7dcbbd964-4x4lf"] Oct 30 00:26:18.329368 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.329331 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c40ae47-fba8-4f1a-a26e-d8ddcfb636af-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-522ea7-7dcbbd964-4x4lf\" (UID: \"3c40ae47-fba8-4f1a-a26e-d8ddcfb636af\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-522ea7-7dcbbd964-4x4lf" Oct 30 00:26:18.329633 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.329614 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkkng\" (UniqueName: \"kubernetes.io/projected/3c40ae47-fba8-4f1a-a26e-d8ddcfb636af-kube-api-access-bkkng\") pod \"payload-processing-e2e-praxis-a-522ea7-7dcbbd964-4x4lf\" (UID: \"3c40ae47-fba8-4f1a-a26e-d8ddcfb636af\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-522ea7-7dcbbd964-4x4lf" Oct 30 00:26:18.455846 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.455711 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c40ae47-fba8-4f1a-a26e-d8ddcfb636af-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-522ea7-7dcbbd964-4x4lf\" (UID: \"3c40ae47-fba8-4f1a-a26e-d8ddcfb636af\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-522ea7-7dcbbd964-4x4lf" Oct 30 00:26:18.456811 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.456328 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkkng\" (UniqueName: \"kubernetes.io/projected/3c40ae47-fba8-4f1a-a26e-d8ddcfb636af-kube-api-access-bkkng\") pod \"payload-processing-e2e-praxis-a-522ea7-7dcbbd964-4x4lf\" (UID: \"3c40ae47-fba8-4f1a-a26e-d8ddcfb636af\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-522ea7-7dcbbd964-4x4lf" Oct 30 00:26:18.459635 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.459510 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c40ae47-fba8-4f1a-a26e-d8ddcfb636af-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-522ea7-7dcbbd964-4x4lf\" (UID: \"3c40ae47-fba8-4f1a-a26e-d8ddcfb636af\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-522ea7-7dcbbd964-4x4lf" Oct 30 00:26:18.529259 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.529206 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkkng\" (UniqueName: \"kubernetes.io/projected/3c40ae47-fba8-4f1a-a26e-d8ddcfb636af-kube-api-access-bkkng\") pod \"payload-processing-e2e-praxis-a-522ea7-7dcbbd964-4x4lf\" (UID: \"3c40ae47-fba8-4f1a-a26e-d8ddcfb636af\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-522ea7-7dcbbd964-4x4lf" Oct 30 00:26:18.567982 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:26:18.567906 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod794f4878_3fd8_4f9c_8167_98b6990270cf.slice/crio-286811a3f27eb1c330defe5f726c55b3aad67bff1b4ea052a3538d57ec09b41e WatchSource:0}: Error finding container 286811a3f27eb1c330defe5f726c55b3aad67bff1b4ea052a3538d57ec09b41e: Status 404 returned error can't find the container with id 286811a3f27eb1c330defe5f726c55b3aad67bff1b4ea052a3538d57ec09b41e Oct 30 00:26:18.569575 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.569545 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-522ea7-7dfd54bbf5-2frb5"] Oct 30 00:26:18.587850 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.587786 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-522ea7-7dcbbd964-4x4lf" Oct 30 00:26:18.717539 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.717502 5001 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 00:26:18.717692 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.717568 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 00:26:18.743310 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.742820 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-522ea7-7dfd54bbf5-2frb5" event={"ID":"794f4878-3fd8-4f9c-8167-98b6990270cf","Type":"ContainerStarted","Data":"5f5043770887052e1a06ea5ea8d9be0673de3b269edcb88a61671490126a47d8"} Oct 30 00:26:18.743310 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.742900 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-522ea7-7dfd54bbf5-2frb5" event={"ID":"794f4878-3fd8-4f9c-8167-98b6990270cf","Type":"ContainerStarted","Data":"286811a3f27eb1c330defe5f726c55b3aad67bff1b4ea052a3538d57ec09b41e"} Oct 30 00:26:18.744106 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.744054 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-522ea7-7dfd54bbf5-2frb5" Oct 30 00:26:18.747093 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.747067 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 00:26:18.747221 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.747118 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 00:26:18.788294 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.788171 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-522ea7-7dfd54bbf5-2frb5" podStartSLOduration=1.788156609 podStartE2EDuration="1.788156609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:26:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:26:18.785731291 +0000 UTC m=+1293.991232474" watchObservedRunningTime="2026-10-30 00:26:18.788156609 +0000 UTC m=+1293.993657790" Oct 30 00:26:18.842431 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.842397 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn"] Oct 30 00:26:18.848335 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.848311 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:26:18.863211 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.862583 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-93016b-openshift-default-dockercfg-6ddrc\"" Oct 30 00:26:18.864320 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.864207 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istio-data\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:26:18.864519 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.864372 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istiod-ca-cert\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:26:18.864596 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.864518 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istio-podinfo\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:26:18.864655 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.864553 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-workload-socket\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:26:18.864724 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.864701 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-credential-socket\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:26:18.864846 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.864818 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-workload-certs\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:26:18.865105 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.864972 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istio-envoy\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:26:18.865105 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.865049 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbq45\" (UniqueName: \"kubernetes.io/projected/34da2ec4-9938-4eaf-9845-dc23bcbc761a-kube-api-access-xbq45\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:26:18.865220 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.865107 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istio-token\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:26:18.871787 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.870960 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn"] Oct 30 00:26:18.901997 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.901967 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-522ea7-7dcbbd964-4x4lf"] Oct 30 00:26:18.902483 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:26:18.902454 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c40ae47_fba8_4f1a_a26e_d8ddcfb636af.slice/crio-565c1436c6a11accb51a33f30c133abe80fc2534eaf83ec3afa472d4596ba153 WatchSource:0}: Error finding container 565c1436c6a11accb51a33f30c133abe80fc2534eaf83ec3afa472d4596ba153: Status 404 returned error can't find the container with id 565c1436c6a11accb51a33f30c133abe80fc2534eaf83ec3afa472d4596ba153 Oct 30 00:26:18.966900 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.966859 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istio-data\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:26:18.967072 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.966968 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istiod-ca-cert\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:26:18.967072 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.967022 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istio-podinfo\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:26:18.967072 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.967053 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-workload-socket\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:26:18.967306 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.967115 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-credential-socket\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:26:18.967306 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.967179 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-workload-certs\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:26:18.967306 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.967294 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istio-envoy\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:26:18.967491 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.967317 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istio-data\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:26:18.967491 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.967371 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-workload-socket\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:26:18.967491 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.967407 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbq45\" (UniqueName: \"kubernetes.io/projected/34da2ec4-9938-4eaf-9845-dc23bcbc761a-kube-api-access-xbq45\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:26:18.967661 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.967508 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istio-token\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:26:18.967661 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.967525 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-credential-socket\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:26:18.967661 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.967650 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-workload-certs\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:26:18.968193 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.968045 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istiod-ca-cert\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:26:18.969718 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.969694 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istio-envoy\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:26:18.970300 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.970280 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istio-podinfo\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:26:18.988712 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.988688 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istio-token\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:26:18.990198 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:18.990175 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbq45\" (UniqueName: \"kubernetes.io/projected/34da2ec4-9938-4eaf-9845-dc23bcbc761a-kube-api-access-xbq45\") pod \"e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:26:19.018042 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:19.006348 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 00:26:19.018042 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:19.006411 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 00:26:19.145663 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:19.145620 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0751af9f-vrtt9"] Oct 30 00:26:19.179371 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:19.179335 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:26:19.284376 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:19.284341 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0751af9f-vrtt9"] Oct 30 00:26:19.618945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:19.616553 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:26:19.736846 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:19.734048 5001 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 00:26:19.736846 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:19.734104 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 00:26:19.753432 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:19.743455 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 00:26:19.753432 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:19.743506 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 00:26:19.764769 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:19.764735 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-522ea7-7dcbbd964-4x4lf" event={"ID":"3c40ae47-fba8-4f1a-a26e-d8ddcfb636af","Type":"ContainerStarted","Data":"a7cd42cbd01058795f3409dedc86e931ace97eb08e693c19ff82618f5dfdab1f"} Oct 30 00:26:19.765122 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:19.765085 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-522ea7-7dcbbd964-4x4lf" event={"ID":"3c40ae47-fba8-4f1a-a26e-d8ddcfb636af","Type":"ContainerStarted","Data":"565c1436c6a11accb51a33f30c133abe80fc2534eaf83ec3afa472d4596ba153"} Oct 30 00:26:19.765339 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:19.765323 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-522ea7-7dcbbd964-4x4lf" Oct 30 00:26:19.810282 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:19.810182 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-522ea7-7dcbbd964-4x4lf" podStartSLOduration=2.810168141 podStartE2EDuration="2.810168141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:26:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:26:19.806678612 +0000 UTC m=+1295.012179793" watchObservedRunningTime="2026-10-30 00:26:19.810168141 +0000 UTC m=+1295.015669321" Oct 30 00:26:19.996972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:19.996299 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 00:26:19.996972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:19.996360 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 00:26:20.117113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:20.117076 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn"] Oct 30 00:26:20.132214 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:20.129127 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:26:20.132214 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:20.129247 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:26:20.132214 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:20.129296 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:26:20.717731 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:20.717692 5001 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 00:26:20.718183 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:20.717762 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 00:26:20.742577 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:20.742544 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 00:26:20.742756 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:20.742605 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 00:26:20.778600 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:20.778503 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" event={"ID":"34da2ec4-9938-4eaf-9845-dc23bcbc761a","Type":"ContainerStarted","Data":"74bd4f05adabe0d4f743035fa48fd38389be4628b8c007ae54ce96696cec5737"} Oct 30 00:26:20.778600 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:20.778573 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" event={"ID":"34da2ec4-9938-4eaf-9845-dc23bcbc761a","Type":"ContainerStarted","Data":"a78c6fc9271dfd313e8b4b436ce9cb990c9ceaa587409f5d4edc5f8a71e10f8e"} Oct 30 00:26:20.829583 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:20.829526 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" podStartSLOduration=2.829510042 podStartE2EDuration="2.829510042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:26:20.827409579 +0000 UTC m=+1296.032910759" watchObservedRunningTime="2026-10-30 00:26:20.829510042 +0000 UTC m=+1296.035011224" Oct 30 00:26:20.922065 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:20.922025 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="102fe425-f28e-4191-881e-369a6b35a761" path="/var/lib/kubelet/pods/102fe425-f28e-4191-881e-369a6b35a761/volumes" Oct 30 00:26:20.994938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:20.994823 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:26:20.995100 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:20.995041 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 00:26:20.995195 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:20.995093 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 00:26:21.183725 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:21.183694 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:26:21.186941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:21.185683 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 00:26:21.186941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:21.185753 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" podUID="34da2ec4-9938-4eaf-9845-dc23bcbc761a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 00:26:21.717487 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:21.717443 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:26:21.717748 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:21.717723 5001 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 00:26:21.718198 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:21.717774 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 00:26:21.743082 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:21.743058 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 00:26:21.743188 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:21.743101 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 00:26:21.995378 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:21.995302 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 00:26:21.995378 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:21.995356 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 00:26:22.180690 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:22.180656 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 00:26:22.180856 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:22.180708 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" podUID="34da2ec4-9938-4eaf-9845-dc23bcbc761a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 00:26:22.717267 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:22.717230 5001 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 00:26:22.717429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:22.717285 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 00:26:22.742613 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:22.742578 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 00:26:22.743019 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:22.742629 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 00:26:22.995641 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:22.995559 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 00:26:22.995641 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:22.995612 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 00:26:23.181091 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:23.181052 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 00:26:23.181256 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:23.181110 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" podUID="34da2ec4-9938-4eaf-9845-dc23bcbc761a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 00:26:23.732046 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:23.727105 5001 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 00:26:23.732046 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:23.727207 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 00:26:23.754937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:23.745744 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 00:26:23.754937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:23.745806 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 00:26:24.009138 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:24.009053 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 00:26:24.009392 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:24.009362 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 00:26:24.180677 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:24.180640 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 00:26:24.180857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:24.180690 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" podUID="34da2ec4-9938-4eaf-9845-dc23bcbc761a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 00:26:24.717497 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:24.717465 5001 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 00:26:24.717688 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:24.717519 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 00:26:24.742964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:24.742925 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 00:26:24.743151 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:24.743001 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 00:26:24.995740 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:24.995663 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 00:26:24.995740 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:24.995712 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 00:26:25.223935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:25.216082 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 00:26:25.223935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:25.216145 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" podUID="34da2ec4-9938-4eaf-9845-dc23bcbc761a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 00:26:25.717772 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:25.717736 5001 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 00:26:25.717975 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:25.717788 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 00:26:25.743018 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:25.742987 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 00:26:25.743167 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:25.743040 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 00:26:25.995731 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:25.995624 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 00:26:25.995731 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:25.995680 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 00:26:26.180830 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:26.180789 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 00:26:26.181034 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:26.180848 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" podUID="34da2ec4-9938-4eaf-9845-dc23bcbc761a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 00:26:26.724331 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:26.722033 5001 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 00:26:26.724331 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:26.722090 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 00:26:26.742821 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:26.742794 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 00:26:26.743060 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:26.743040 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 00:26:27.000064 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:26.999993 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:26:27.001118 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:27.001097 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:26:27.186995 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:27.186958 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:26:27.187693 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:27.187657 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:26:27.188437 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:27.188419 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:26:27.737058 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:27.737016 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:26:27.738225 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:27.738200 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:26:27.743077 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:27.743055 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 00:26:27.743175 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:27.743103 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 00:26:28.742459 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:28.742417 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 00:26:28.742858 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:28.742484 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 00:26:29.824253 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:29.824215 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 00:26:29.824986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:29.824949 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 00:26:30.197481 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.197408 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw"] Oct 30 00:26:30.203490 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.203465 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:26:30.221371 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.221338 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a15230-openshift-default-dockercfg-2bt7v\"" Oct 30 00:26:30.259485 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.259399 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw"] Oct 30 00:26:30.298352 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.298316 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-credential-socket\") pod \"e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:26:30.298559 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.298429 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/809701ad-afba-418f-8844-36e0baa399df-istio-podinfo\") pod \"e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:26:30.298559 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.298541 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/809701ad-afba-418f-8844-36e0baa399df-istio-token\") pod \"e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:26:30.298700 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.298585 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/809701ad-afba-418f-8844-36e0baa399df-istiod-ca-cert\") pod \"e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:26:30.298700 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.298683 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-workload-socket\") pod \"e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:26:30.298782 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.298751 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-istio-envoy\") pod \"e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:26:30.298945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.298906 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-istio-data\") pod \"e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:26:30.299049 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.299026 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbwwn\" (UniqueName: \"kubernetes.io/projected/809701ad-afba-418f-8844-36e0baa399df-kube-api-access-tbwwn\") pod \"e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:26:30.299114 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.299100 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-workload-certs\") pod \"e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:26:30.411501 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.409734 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-istio-envoy\") pod \"e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:26:30.411501 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.409837 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-istio-data\") pod \"e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:26:30.411501 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.409945 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbwwn\" (UniqueName: \"kubernetes.io/projected/809701ad-afba-418f-8844-36e0baa399df-kube-api-access-tbwwn\") pod \"e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:26:30.411501 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.410014 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-workload-certs\") pod \"e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:26:30.411501 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.410080 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-credential-socket\") pod \"e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:26:30.411501 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.410167 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/809701ad-afba-418f-8844-36e0baa399df-istio-podinfo\") pod \"e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:26:30.411501 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.410307 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/809701ad-afba-418f-8844-36e0baa399df-istio-token\") pod \"e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:26:30.411501 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.410364 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/809701ad-afba-418f-8844-36e0baa399df-istiod-ca-cert\") pod \"e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:26:30.411501 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.410499 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-workload-socket\") pod \"e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:26:30.411501 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.411077 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-workload-socket\") pod \"e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:26:30.423934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.412520 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-credential-socket\") pod \"e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:26:30.423934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.412790 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-istio-data\") pod \"e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:26:30.423934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.413343 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-workload-certs\") pod \"e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:26:30.424668 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.424607 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/809701ad-afba-418f-8844-36e0baa399df-istiod-ca-cert\") pod \"e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:26:30.459615 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.459504 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-istio-envoy\") pod \"e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:26:30.476935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.464686 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbwwn\" (UniqueName: \"kubernetes.io/projected/809701ad-afba-418f-8844-36e0baa399df-kube-api-access-tbwwn\") pod \"e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:26:30.494243 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.488253 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/809701ad-afba-418f-8844-36e0baa399df-istio-podinfo\") pod \"e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:26:30.505578 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.505511 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/809701ad-afba-418f-8844-36e0baa399df-istio-token\") pod \"e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:26:30.558364 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.558309 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:26:30.776937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.768119 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 00:26:30.776937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.768205 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 00:26:30.854459 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.854428 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-522ea7-7dfd54bbf5-2frb5" Oct 30 00:26:30.857706 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:30.856747 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-522ea7-7dcbbd964-4x4lf" Oct 30 00:26:31.392342 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:31.392299 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:26:31.429280 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:31.429230 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:26:31.789952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:31.787272 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 00:26:31.789952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:31.787351 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 00:26:32.123146 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:26:32.118017 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod809701ad_afba_418f_8844_36e0baa399df.slice/crio-6966b0a85c287169ee050052aea8f7d4b5d9d9d30a09a54a1bdb4f215d5c0bd1 WatchSource:0}: Error finding container 6966b0a85c287169ee050052aea8f7d4b5d9d9d30a09a54a1bdb4f215d5c0bd1: Status 404 returned error can't find the container with id 6966b0a85c287169ee050052aea8f7d4b5d9d9d30a09a54a1bdb4f215d5c0bd1 Oct 30 00:26:32.149944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:32.133011 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw"] Oct 30 00:26:32.149944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:32.136280 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:26:32.149944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:32.136403 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:26:32.149944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:32.136453 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:26:32.341934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:32.340831 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" event={"ID":"809701ad-afba-418f-8844-36e0baa399df","Type":"ContainerStarted","Data":"05dded4bfe99293e12b3924e62f99c1ecd6e4749594d2d1d2a5be65e83171360"} Oct 30 00:26:32.341934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:32.340924 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" event={"ID":"809701ad-afba-418f-8844-36e0baa399df","Type":"ContainerStarted","Data":"6966b0a85c287169ee050052aea8f7d4b5d9d9d30a09a54a1bdb4f215d5c0bd1"} Oct 30 00:26:32.384994 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:32.384503 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podStartSLOduration=2.384482302 podStartE2EDuration="2.384482302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:26:32.381297396 +0000 UTC m=+1307.586798579" watchObservedRunningTime="2026-10-30 00:26:32.384482302 +0000 UTC m=+1307.589983480" Oct 30 00:26:32.562963 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:32.562262 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:26:32.746047 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:32.744419 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 00:26:32.746047 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:32.744475 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 00:26:33.560506 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:33.560466 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:26:33.569044 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:33.568995 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:26:33.747665 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:33.747631 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:26:33.748857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:33.748832 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:26:34.559283 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:34.559250 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:26:34.559435 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:34.559302 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:26:35.559999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:35.559965 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:26:35.560404 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:35.560019 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:26:36.559177 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:36.559142 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:26:36.559433 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:36.559195 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:26:37.559337 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:37.559300 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:26:37.563650 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:37.563606 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:26:38.562896 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:38.559509 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:26:38.563737 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:38.563696 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:26:39.031599 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:39.031568 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7afafb7b-xg76q"] Oct 30 00:26:39.040187 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:39.040159 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7afafb7b-xg76q"] Oct 30 00:26:39.559065 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:39.559026 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:26:39.559267 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:39.559088 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:26:40.559762 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:40.559729 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:26:40.560204 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:40.559793 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:26:40.920295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:40.920201 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aae7ddca-8ecc-4d30-809a-23b2b39b506b" path="/var/lib/kubelet/pods/aae7ddca-8ecc-4d30-809a-23b2b39b506b/volumes" Oct 30 00:26:41.559697 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:41.559663 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:26:41.559888 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:41.559725 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:26:42.559509 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:42.559476 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:26:42.559692 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:42.559525 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:26:43.559422 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:43.559389 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:26:43.559852 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:43.559467 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:26:44.564935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:44.564335 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:26:44.564935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:44.564398 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:26:45.558738 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:45.558703 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:26:45.558963 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:45.558929 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:26:45.559039 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:45.558979 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:26:46.559514 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:46.559474 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:26:46.559901 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:46.559540 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:26:47.382128 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:26:47.381841 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:26:47.559159 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:47.559126 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:26:47.559328 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:47.559180 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:26:48.560041 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:48.560004 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:26:48.560469 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:48.560065 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:26:49.559474 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:49.559429 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:26:49.559698 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:49.559499 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:26:50.559874 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:50.559842 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:26:50.560285 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:50.559889 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:26:51.559121 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:51.559085 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:26:51.559309 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:51.559142 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:26:52.107895 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:52.107853 5001 scope.go:122] "RemoveContainer" containerID="0f0efa84858f2ffee9c9b9e2c078265ba81920ca508a5984030947c68ce7b87e" Oct 30 00:26:52.129226 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:52.129198 5001 scope.go:122] "RemoveContainer" containerID="11b033b2d8b2d3fa4ff82d0a7858e42f5002a34c34490bbe6a38b0090eeeeea4" Oct 30 00:26:52.148566 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:52.148531 5001 scope.go:122] "RemoveContainer" containerID="4017a53718d7ee53742f39aed265005721fcc99d076d9036815ca7451e243975" Oct 30 00:26:52.208062 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:52.208033 5001 scope.go:122] "RemoveContainer" containerID="ddf487abfd86f974827726f1c8f710c14f4fb25ffd16efab8fc46f4fc756a1e3" Oct 30 00:26:52.227140 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:52.227118 5001 scope.go:122] "RemoveContainer" containerID="0cb53eeae4bd1e5ed7506b877b0d496b6c68857efd850524a4395afa1bba4f30" Oct 30 00:26:52.251282 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:52.251253 5001 scope.go:122] "RemoveContainer" containerID="35120da54338ba6c0152436ae97981a7ee9c2a955bd4bcd53d920db4ac4f8ad2" Oct 30 00:26:52.334326 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:52.329767 5001 scope.go:122] "RemoveContainer" containerID="eb10c274bdbdf37c4c8c47cbff0c1eb4ac02a310ba576905d1483d2cd0ac0b15" Oct 30 00:26:52.353493 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:52.353351 5001 scope.go:122] "RemoveContainer" containerID="6804f1981876a6b16fb1d607ef3b306f901c947c9f5f2dd6428d0adac5b71ab0" Oct 30 00:26:52.559669 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:52.559634 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:26:52.559856 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:52.559684 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:26:53.559784 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:53.559744 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:26:53.560213 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:53.559814 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:26:54.559316 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:54.559285 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:26:54.559494 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:54.559346 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:26:55.569945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:55.562807 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:26:55.569945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:55.562884 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:26:56.559569 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:56.559534 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:26:56.559736 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:56.559586 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:26:56.600185 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:56.600144 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd"] Oct 30 00:26:56.600644 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:56.600338 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" podUID="9acc881e-944d-4cdb-a67c-86c87d85d9e5" containerName="istio-proxy" containerID="cri-o://18e650f0ad0b3bf970b283925419150dec9a11eb5a4585b0f75b3b0b54756f74" gracePeriod=30 Oct 30 00:26:56.610990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:56.610474 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s"] Oct 30 00:26:56.773888 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:56.773842 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" podUID="257eedd7-4d1b-4e1f-91c5-b5af649dc927" containerName="istio-proxy" containerID="cri-o://b9e8adf6739248ea3335007c7091520c85e2c3957c53847b7b595612c8b0ab22" gracePeriod=30 Oct 30 00:26:57.559630 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:57.559594 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:26:57.559814 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:57.559646 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:26:58.032767 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.030847 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf"] Oct 30 00:26:58.041088 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.041053 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:26:58.054752 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.054724 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-43dc72-openshift-default-dockercfg-bvk8c\"" Oct 30 00:26:58.077202 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.077164 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-workload-certs\") pod \"e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:26:58.077387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.077319 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-istio-envoy\") pod \"e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:26:58.077387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.077353 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe41eb3-2947-447a-aa04-4eca33589f1d-istio-token\") pod \"e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:26:58.077530 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.077396 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe41eb3-2947-447a-aa04-4eca33589f1d-istiod-ca-cert\") pod \"e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:26:58.077530 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.077442 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-workload-socket\") pod \"e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:26:58.077717 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.077694 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-istio-data\") pod \"e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:26:58.077805 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.077782 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dbgg\" (UniqueName: \"kubernetes.io/projected/dfe41eb3-2947-447a-aa04-4eca33589f1d-kube-api-access-6dbgg\") pod \"e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:26:58.077869 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.077834 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe41eb3-2947-447a-aa04-4eca33589f1d-istio-podinfo\") pod \"e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:26:58.077953 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.077878 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-credential-socket\") pod \"e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:26:58.083312 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.083286 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf"] Oct 30 00:26:58.189936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.189702 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe41eb3-2947-447a-aa04-4eca33589f1d-istio-podinfo\") pod \"e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:26:58.189936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.189778 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-credential-socket\") pod \"e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:26:58.189936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.189889 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-workload-certs\") pod \"e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:26:58.190240 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.190042 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-istio-envoy\") pod \"e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:26:58.190240 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.190081 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe41eb3-2947-447a-aa04-4eca33589f1d-istio-token\") pod \"e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:26:58.190240 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.190132 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe41eb3-2947-447a-aa04-4eca33589f1d-istiod-ca-cert\") pod \"e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:26:58.190240 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.190176 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-workload-socket\") pod \"e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:26:58.190448 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.190327 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-istio-data\") pod \"e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:26:58.190448 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.190407 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dbgg\" (UniqueName: \"kubernetes.io/projected/dfe41eb3-2947-447a-aa04-4eca33589f1d-kube-api-access-6dbgg\") pod \"e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:26:58.216976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.193990 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-credential-socket\") pod \"e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:26:58.216976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.194237 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-workload-certs\") pod \"e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:26:58.216976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.194791 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-workload-socket\") pod \"e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:26:58.216976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.195502 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe41eb3-2947-447a-aa04-4eca33589f1d-istiod-ca-cert\") pod \"e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:26:58.216976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.198183 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-istio-data\") pod \"e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:26:58.248933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.246608 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe41eb3-2947-447a-aa04-4eca33589f1d-istio-podinfo\") pod \"e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:26:58.248933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.246819 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-istio-envoy\") pod \"e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:26:58.299064 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.262680 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe41eb3-2947-447a-aa04-4eca33589f1d-istio-token\") pod \"e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:26:58.309941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.308571 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dbgg\" (UniqueName: \"kubernetes.io/projected/dfe41eb3-2947-447a-aa04-4eca33589f1d-kube-api-access-6dbgg\") pod \"e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:26:58.425934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.409231 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:26:58.565456 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.565362 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:26:58.565711 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:58.565682 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:26:59.136962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:59.136568 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf"] Oct 30 00:26:59.137386 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:26:59.137205 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe41eb3_2947_447a_aa04_4eca33589f1d.slice/crio-b06df223e7d09ecc405586ad9827e0182406447cb7b34cba192bbcb705d0a10c WatchSource:0}: Error finding container b06df223e7d09ecc405586ad9827e0182406447cb7b34cba192bbcb705d0a10c: Status 404 returned error can't find the container with id b06df223e7d09ecc405586ad9827e0182406447cb7b34cba192bbcb705d0a10c Oct 30 00:26:59.141084 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:59.141039 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:26:59.141204 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:59.141135 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:26:59.141204 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:59.141177 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:26:59.559610 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:59.559573 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:26:59.559804 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:59.559632 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:26:59.899416 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:59.899326 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" event={"ID":"dfe41eb3-2947-447a-aa04-4eca33589f1d","Type":"ContainerStarted","Data":"1604d7cfa69887eecbbdf6d7f18b6f4d47c680b9ef80fce3e6478b12a59d7423"} Oct 30 00:26:59.899416 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:59.899389 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" event={"ID":"dfe41eb3-2947-447a-aa04-4eca33589f1d","Type":"ContainerStarted","Data":"b06df223e7d09ecc405586ad9827e0182406447cb7b34cba192bbcb705d0a10c"} Oct 30 00:26:59.957603 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:26:59.957543 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" podStartSLOduration=1.957523778 podStartE2EDuration="1.957523778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:26:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:26:59.949013994 +0000 UTC m=+1335.154515176" watchObservedRunningTime="2026-10-30 00:26:59.957523778 +0000 UTC m=+1335.163024960" Oct 30 00:27:00.428936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:00.425520 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:27:00.446868 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:00.446831 5001 patch_prober.go:29] interesting pod/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 00:27:00.447678 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:00.447645 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" podUID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 00:27:00.561294 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:00.561241 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:27:00.561557 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:00.561530 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:27:01.417414 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:01.417359 5001 patch_prober.go:29] interesting pod/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 00:27:01.417705 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:01.417677 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" podUID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 00:27:01.560590 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:01.560553 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:27:01.560994 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:01.560610 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:27:01.928144 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:01.928117 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:27:02.058626 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.058601 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istio-data\") pod \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " Oct 30 00:27:02.058758 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.058645 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-workload-socket\") pod \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " Oct 30 00:27:02.058758 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.058683 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istio-envoy\") pod \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " Oct 30 00:27:02.058882 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.058774 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istio-token\" (UniqueName: \"kubernetes.io/projected/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istio-token\") pod \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " Oct 30 00:27:02.058958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.058872 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istio-data" pod "9acc881e-944d-4cdb-a67c-86c87d85d9e5" (UID: "9acc881e-944d-4cdb-a67c-86c87d85d9e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:02.058958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.058904 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istiod-ca-cert\") pod \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " Oct 30 00:27:02.059120 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.059105 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-credential-socket\") pod \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " Oct 30 00:27:02.059205 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.059191 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9acc881e-944d-4cdb-a67c-86c87d85d9e5-kube-api-access-fx946\" (UniqueName: \"kubernetes.io/projected/9acc881e-944d-4cdb-a67c-86c87d85d9e5-kube-api-access-fx946\") pod \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " Oct 30 00:27:02.059296 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.059283 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-workload-certs\") pod \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " Oct 30 00:27:02.059349 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.059309 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-workload-socket" pod "9acc881e-944d-4cdb-a67c-86c87d85d9e5" (UID: "9acc881e-944d-4cdb-a67c-86c87d85d9e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:02.059349 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.059341 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istio-podinfo\") pod \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\" (UID: \"9acc881e-944d-4cdb-a67c-86c87d85d9e5\") " Oct 30 00:27:02.059546 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.059525 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-credential-socket" pod "9acc881e-944d-4cdb-a67c-86c87d85d9e5" (UID: "9acc881e-944d-4cdb-a67c-86c87d85d9e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:02.060001 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.059970 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istiod-ca-cert" pod "9acc881e-944d-4cdb-a67c-86c87d85d9e5" (UID: "9acc881e-944d-4cdb-a67c-86c87d85d9e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:27:02.060288 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.060269 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:02.060377 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.060293 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:02.060377 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.060311 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:02.060377 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.060325 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:02.060649 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.060622 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-workload-certs" pod "9acc881e-944d-4cdb-a67c-86c87d85d9e5" (UID: "9acc881e-944d-4cdb-a67c-86c87d85d9e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:02.062259 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.062229 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istio-token" pod "9acc881e-944d-4cdb-a67c-86c87d85d9e5" (UID: "9acc881e-944d-4cdb-a67c-86c87d85d9e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:27:02.062370 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.062261 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istio-envoy" pod "9acc881e-944d-4cdb-a67c-86c87d85d9e5" (UID: "9acc881e-944d-4cdb-a67c-86c87d85d9e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:02.062611 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.062593 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9acc881e-944d-4cdb-a67c-86c87d85d9e5-kube-api-access-fx946" pod "9acc881e-944d-4cdb-a67c-86c87d85d9e5" (UID: "9acc881e-944d-4cdb-a67c-86c87d85d9e5"). InnerVolumeSpecName "kube-api-access-fx946". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:27:02.062751 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.062732 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istio-podinfo" pod "9acc881e-944d-4cdb-a67c-86c87d85d9e5" (UID: "9acc881e-944d-4cdb-a67c-86c87d85d9e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:27:02.071887 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.071860 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:27:02.159414 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.159386 5001 generic.go:356] "Generic (PLEG): container finished" podID="9acc881e-944d-4cdb-a67c-86c87d85d9e5" containerID="18e650f0ad0b3bf970b283925419150dec9a11eb5a4585b0f75b3b0b54756f74" exitCode=0 Oct 30 00:27:02.159579 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.159432 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" Oct 30 00:27:02.159579 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.159465 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" event={"ID":"9acc881e-944d-4cdb-a67c-86c87d85d9e5","Type":"ContainerDied","Data":"18e650f0ad0b3bf970b283925419150dec9a11eb5a4585b0f75b3b0b54756f74"} Oct 30 00:27:02.159579 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.159518 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd" event={"ID":"9acc881e-944d-4cdb-a67c-86c87d85d9e5","Type":"ContainerDied","Data":"c57f945bba2b27aab3e100310355f32502beea29a813b2ea53ffd1ab82da5743"} Oct 30 00:27:02.159579 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.159539 5001 scope.go:122] "RemoveContainer" containerID="18e650f0ad0b3bf970b283925419150dec9a11eb5a4585b0f75b3b0b54756f74" Oct 30 00:27:02.160794 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.160776 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istio-data\") pod \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " Oct 30 00:27:02.160898 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.160822 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istio-envoy\") pod \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " Oct 30 00:27:02.160898 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.160851 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istiod-ca-cert\") pod \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " Oct 30 00:27:02.161035 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.160982 5001 generic.go:356] "Generic (PLEG): container finished" podID="257eedd7-4d1b-4e1f-91c5-b5af649dc927" containerID="b9e8adf6739248ea3335007c7091520c85e2c3957c53847b7b595612c8b0ab22" exitCode=0 Oct 30 00:27:02.161035 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.160993 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-credential-socket\") pod \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " Oct 30 00:27:02.161143 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.161042 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istio-podinfo\") pod \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " Oct 30 00:27:02.161143 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.161108 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-workload-socket\") pod \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " Oct 30 00:27:02.161241 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.161123 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istio-data" pod "257eedd7-4d1b-4e1f-91c5-b5af649dc927" (UID: "257eedd7-4d1b-4e1f-91c5-b5af649dc927"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:02.161241 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.161159 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-workload-certs\") pod \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " Oct 30 00:27:02.161342 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.161284 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istiod-ca-cert" pod "257eedd7-4d1b-4e1f-91c5-b5af649dc927" (UID: "257eedd7-4d1b-4e1f-91c5-b5af649dc927"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:27:02.161342 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.161291 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istio-token\" (UniqueName: \"kubernetes.io/projected/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istio-token\") pod \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " Oct 30 00:27:02.161451 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.161399 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/257eedd7-4d1b-4e1f-91c5-b5af649dc927-kube-api-access-f8t9n\" (UniqueName: \"kubernetes.io/projected/257eedd7-4d1b-4e1f-91c5-b5af649dc927-kube-api-access-f8t9n\") pod \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\" (UID: \"257eedd7-4d1b-4e1f-91c5-b5af649dc927\") " Oct 30 00:27:02.161943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.161904 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" Oct 30 00:27:02.162112 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.162070 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" event={"ID":"257eedd7-4d1b-4e1f-91c5-b5af649dc927","Type":"ContainerDied","Data":"b9e8adf6739248ea3335007c7091520c85e2c3957c53847b7b595612c8b0ab22"} Oct 30 00:27:02.162112 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.162109 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s" event={"ID":"257eedd7-4d1b-4e1f-91c5-b5af649dc927","Type":"ContainerDied","Data":"2d309257816785c4dff62b3c8de05ef3d966a83e09d054b73f9fc86f39977e8b"} Oct 30 00:27:02.162348 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.162325 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-workload-socket" pod "257eedd7-4d1b-4e1f-91c5-b5af649dc927" (UID: "257eedd7-4d1b-4e1f-91c5-b5af649dc927"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:02.162436 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.162400 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx946\" (UniqueName: \"kubernetes.io/projected/9acc881e-944d-4cdb-a67c-86c87d85d9e5-kube-api-access-fx946\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:02.162436 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.162426 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:02.162596 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.162441 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:02.162596 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.162454 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:02.162596 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.162467 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:02.162596 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.162479 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:02.162596 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.162492 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9acc881e-944d-4cdb-a67c-86c87d85d9e5-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:02.162596 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.162505 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-workload-certs" pod "257eedd7-4d1b-4e1f-91c5-b5af649dc927" (UID: "257eedd7-4d1b-4e1f-91c5-b5af649dc927"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:02.163869 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.163844 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istio-token" pod "257eedd7-4d1b-4e1f-91c5-b5af649dc927" (UID: "257eedd7-4d1b-4e1f-91c5-b5af649dc927"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:27:02.164836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.164810 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istio-podinfo" pod "257eedd7-4d1b-4e1f-91c5-b5af649dc927" (UID: "257eedd7-4d1b-4e1f-91c5-b5af649dc927"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:27:02.165004 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.164981 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/257eedd7-4d1b-4e1f-91c5-b5af649dc927-kube-api-access-f8t9n" pod "257eedd7-4d1b-4e1f-91c5-b5af649dc927" (UID: "257eedd7-4d1b-4e1f-91c5-b5af649dc927"). InnerVolumeSpecName "kube-api-access-f8t9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:27:02.165057 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.165022 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-credential-socket" pod "257eedd7-4d1b-4e1f-91c5-b5af649dc927" (UID: "257eedd7-4d1b-4e1f-91c5-b5af649dc927"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:02.166875 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.166845 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istio-envoy" pod "257eedd7-4d1b-4e1f-91c5-b5af649dc927" (UID: "257eedd7-4d1b-4e1f-91c5-b5af649dc927"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:02.172710 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.172690 5001 scope.go:122] "RemoveContainer" containerID="18e650f0ad0b3bf970b283925419150dec9a11eb5a4585b0f75b3b0b54756f74" Oct 30 00:27:02.173034 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:27:02.173015 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18e650f0ad0b3bf970b283925419150dec9a11eb5a4585b0f75b3b0b54756f74\": container with ID starting with 18e650f0ad0b3bf970b283925419150dec9a11eb5a4585b0f75b3b0b54756f74 not found: ID does not exist" containerID="18e650f0ad0b3bf970b283925419150dec9a11eb5a4585b0f75b3b0b54756f74" Oct 30 00:27:02.173102 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.173043 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18e650f0ad0b3bf970b283925419150dec9a11eb5a4585b0f75b3b0b54756f74"} err="failed to get container status \"18e650f0ad0b3bf970b283925419150dec9a11eb5a4585b0f75b3b0b54756f74\": rpc error: code = NotFound desc = could not find container \"18e650f0ad0b3bf970b283925419150dec9a11eb5a4585b0f75b3b0b54756f74\": container with ID starting with 18e650f0ad0b3bf970b283925419150dec9a11eb5a4585b0f75b3b0b54756f74 not found: ID does not exist" Oct 30 00:27:02.173102 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.173061 5001 scope.go:122] "RemoveContainer" containerID="b9e8adf6739248ea3335007c7091520c85e2c3957c53847b7b595612c8b0ab22" Oct 30 00:27:02.184354 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.184338 5001 scope.go:122] "RemoveContainer" containerID="b9e8adf6739248ea3335007c7091520c85e2c3957c53847b7b595612c8b0ab22" Oct 30 00:27:02.184598 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:27:02.184578 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9e8adf6739248ea3335007c7091520c85e2c3957c53847b7b595612c8b0ab22\": container with ID starting with b9e8adf6739248ea3335007c7091520c85e2c3957c53847b7b595612c8b0ab22 not found: ID does not exist" containerID="b9e8adf6739248ea3335007c7091520c85e2c3957c53847b7b595612c8b0ab22" Oct 30 00:27:02.184651 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.184604 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9e8adf6739248ea3335007c7091520c85e2c3957c53847b7b595612c8b0ab22"} err="failed to get container status \"b9e8adf6739248ea3335007c7091520c85e2c3957c53847b7b595612c8b0ab22\": rpc error: code = NotFound desc = could not find container \"b9e8adf6739248ea3335007c7091520c85e2c3957c53847b7b595612c8b0ab22\": container with ID starting with b9e8adf6739248ea3335007c7091520c85e2c3957c53847b7b595612c8b0ab22 not found: ID does not exist" Oct 30 00:27:02.212079 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.212049 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd"] Oct 30 00:27:02.217085 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.217066 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-7gwcd"] Oct 30 00:27:02.268940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.265650 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:02.268940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.265689 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:02.268940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.265704 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:02.268940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.265724 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:02.268940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.265745 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/257eedd7-4d1b-4e1f-91c5-b5af649dc927-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:02.268940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.265765 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/257eedd7-4d1b-4e1f-91c5-b5af649dc927-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:02.268940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.265783 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8t9n\" (UniqueName: \"kubernetes.io/projected/257eedd7-4d1b-4e1f-91c5-b5af649dc927-kube-api-access-f8t9n\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:02.416471 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.416392 5001 patch_prober.go:29] interesting pod/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 00:27:02.416471 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.416455 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" podUID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 00:27:02.511922 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.511891 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s"] Oct 30 00:27:02.517029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.517002 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-2kl8s"] Oct 30 00:27:02.559185 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.559159 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:27:02.559286 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.559212 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:27:02.559286 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.559252 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:27:02.559831 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.559807 5001 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="istio-proxy" containerStatusID={"Type":"cri-o","ID":"05dded4bfe99293e12b3924e62f99c1ecd6e4749594d2d1d2a5be65e83171360"} pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" containerMessage="Container istio-proxy failed startup probe, will be restarted" Oct 30 00:27:02.559944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.559852 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" containerID="cri-o://05dded4bfe99293e12b3924e62f99c1ecd6e4749594d2d1d2a5be65e83171360" gracePeriod=30 Oct 30 00:27:02.919904 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.919868 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="257eedd7-4d1b-4e1f-91c5-b5af649dc927" path="/var/lib/kubelet/pods/257eedd7-4d1b-4e1f-91c5-b5af649dc927/volumes" Oct 30 00:27:02.920489 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:02.920471 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9acc881e-944d-4cdb-a67c-86c87d85d9e5" path="/var/lib/kubelet/pods/9acc881e-944d-4cdb-a67c-86c87d85d9e5/volumes" Oct 30 00:27:03.417091 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:03.417055 5001 patch_prober.go:29] interesting pod/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 00:27:03.417269 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:03.417117 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" podUID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 00:27:03.941965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:03.940468 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl"] Oct 30 00:27:03.941965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:03.941737 5001 cpu_manager.go:414] "containerMap: removing container" podUID="257eedd7-4d1b-4e1f-91c5-b5af649dc927" containerName="istio-proxy" Oct 30 00:27:03.941965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:03.941758 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="257eedd7-4d1b-4e1f-91c5-b5af649dc927" containerName="istio-proxy" Oct 30 00:27:03.941965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:03.941784 5001 cpu_manager.go:414] "containerMap: removing container" podUID="9acc881e-944d-4cdb-a67c-86c87d85d9e5" containerName="istio-proxy" Oct 30 00:27:03.941965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:03.941792 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9acc881e-944d-4cdb-a67c-86c87d85d9e5" containerName="istio-proxy" Oct 30 00:27:03.941965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:03.941966 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="257eedd7-4d1b-4e1f-91c5-b5af649dc927" containerName="istio-proxy" Oct 30 00:27:03.942696 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:03.941983 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9acc881e-944d-4cdb-a67c-86c87d85d9e5" containerName="istio-proxy" Oct 30 00:27:03.948222 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:03.948195 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:27:03.961748 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:03.961721 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-00df5f-openshift-default-dockercfg-vn4q5\"" Oct 30 00:27:03.978234 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:03.978195 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl"] Oct 30 00:27:04.119408 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.119343 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-credential-socket\") pod \"e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:27:04.119684 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.119658 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-workload-certs\") pod \"e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:27:04.119865 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.119848 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a684c9be-f1fe-420b-a3a0-16ec60772131-istiod-ca-cert\") pod \"e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:27:04.120038 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.120021 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a684c9be-f1fe-420b-a3a0-16ec60772131-istio-podinfo\") pod \"e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:27:04.120294 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.120274 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a684c9be-f1fe-420b-a3a0-16ec60772131-istio-token\") pod \"e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:27:04.120469 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.120449 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kl5h\" (UniqueName: \"kubernetes.io/projected/a684c9be-f1fe-420b-a3a0-16ec60772131-kube-api-access-6kl5h\") pod \"e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:27:04.120603 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.120589 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-istio-envoy\") pod \"e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:27:04.120713 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.120697 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-istio-data\") pod \"e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:27:04.120976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.120958 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-workload-socket\") pod \"e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:27:04.204014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.203935 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-43dc72-5f4bd85fdc-2fn22"] Oct 30 00:27:04.208623 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.208606 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-43dc72-5f4bd85fdc-2fn22" Oct 30 00:27:04.222350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.222321 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-istio-envoy\") pod \"e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:27:04.222520 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.222367 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-istio-data\") pod \"e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:27:04.222520 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.222505 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-workload-socket\") pod \"e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:27:04.222672 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.222650 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-credential-socket\") pod \"e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:27:04.222741 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.222681 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-workload-certs\") pod \"e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:27:04.222741 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.222720 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-43dc72\"" Oct 30 00:27:04.222878 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.222854 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-istio-data\") pod \"e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:27:04.222979 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.222736 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a684c9be-f1fe-420b-a3a0-16ec60772131-istiod-ca-cert\") pod \"e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:27:04.223054 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.222984 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-workload-socket\") pod \"e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:27:04.223054 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.223010 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a684c9be-f1fe-420b-a3a0-16ec60772131-istio-podinfo\") pod \"e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:27:04.223165 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.223113 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a684c9be-f1fe-420b-a3a0-16ec60772131-istio-token\") pod \"e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:27:04.223165 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.223152 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-43dc72-dockercfg-j8dvs\"" Oct 30 00:27:04.223266 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.223188 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kl5h\" (UniqueName: \"kubernetes.io/projected/a684c9be-f1fe-420b-a3a0-16ec60772131-kube-api-access-6kl5h\") pod \"e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:27:04.223426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.223403 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a684c9be-f1fe-420b-a3a0-16ec60772131-istiod-ca-cert\") pod \"e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:27:04.223541 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.223403 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-credential-socket\") pod \"e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:27:04.223726 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.223706 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-workload-certs\") pod \"e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:27:04.229067 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.226493 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a684c9be-f1fe-420b-a3a0-16ec60772131-istio-podinfo\") pod \"e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:27:04.234208 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.234182 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-43dc72-5f4bd85fdc-2fn22"] Oct 30 00:27:04.255508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.255472 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-istio-envoy\") pod \"e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:27:04.258982 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.258896 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a684c9be-f1fe-420b-a3a0-16ec60772131-istio-token\") pod \"e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:27:04.261723 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.261694 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kl5h\" (UniqueName: \"kubernetes.io/projected/a684c9be-f1fe-420b-a3a0-16ec60772131-kube-api-access-6kl5h\") pod \"e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:27:04.270947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.267422 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:27:04.325190 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.325146 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddhg2\" (UniqueName: \"kubernetes.io/projected/6c04baf6-1ee8-47ee-b3a7-3f92393b644b-kube-api-access-ddhg2\") pod \"payload-pre-processing-e2e-api-b-43dc72-5f4bd85fdc-2fn22\" (UID: \"6c04baf6-1ee8-47ee-b3a7-3f92393b644b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-43dc72-5f4bd85fdc-2fn22" Oct 30 00:27:04.325405 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.325329 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c04baf6-1ee8-47ee-b3a7-3f92393b644b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-43dc72-5f4bd85fdc-2fn22\" (UID: \"6c04baf6-1ee8-47ee-b3a7-3f92393b644b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-43dc72-5f4bd85fdc-2fn22" Oct 30 00:27:04.445934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.442347 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddhg2\" (UniqueName: \"kubernetes.io/projected/6c04baf6-1ee8-47ee-b3a7-3f92393b644b-kube-api-access-ddhg2\") pod \"payload-pre-processing-e2e-api-b-43dc72-5f4bd85fdc-2fn22\" (UID: \"6c04baf6-1ee8-47ee-b3a7-3f92393b644b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-43dc72-5f4bd85fdc-2fn22" Oct 30 00:27:04.445934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.442550 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c04baf6-1ee8-47ee-b3a7-3f92393b644b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-43dc72-5f4bd85fdc-2fn22\" (UID: \"6c04baf6-1ee8-47ee-b3a7-3f92393b644b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-43dc72-5f4bd85fdc-2fn22" Oct 30 00:27:04.459989 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.447772 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c04baf6-1ee8-47ee-b3a7-3f92393b644b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-43dc72-5f4bd85fdc-2fn22\" (UID: \"6c04baf6-1ee8-47ee-b3a7-3f92393b644b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-43dc72-5f4bd85fdc-2fn22" Oct 30 00:27:04.494937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.491057 5001 patch_prober.go:29] interesting pod/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 00:27:04.494937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.491134 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" podUID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 00:27:04.677972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.666158 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddhg2\" (UniqueName: \"kubernetes.io/projected/6c04baf6-1ee8-47ee-b3a7-3f92393b644b-kube-api-access-ddhg2\") pod \"payload-pre-processing-e2e-api-b-43dc72-5f4bd85fdc-2fn22\" (UID: \"6c04baf6-1ee8-47ee-b3a7-3f92393b644b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-43dc72-5f4bd85fdc-2fn22" Oct 30 00:27:04.827734 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:04.827695 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-43dc72-5f4bd85fdc-2fn22" Oct 30 00:27:05.133490 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:05.131738 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl"] Oct 30 00:27:05.133490 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:27:05.132021 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda684c9be_f1fe_420b_a3a0_16ec60772131.slice/crio-2c93fa7a05592e1352a687f981d2bca8622191545b10ddd0e87d79e7528b5ee1 WatchSource:0}: Error finding container 2c93fa7a05592e1352a687f981d2bca8622191545b10ddd0e87d79e7528b5ee1: Status 404 returned error can't find the container with id 2c93fa7a05592e1352a687f981d2bca8622191545b10ddd0e87d79e7528b5ee1 Oct 30 00:27:05.159936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:05.155073 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:27:05.159936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:05.155185 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:27:05.159936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:05.155268 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:27:05.281361 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:05.281214 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" event={"ID":"a684c9be-f1fe-420b-a3a0-16ec60772131","Type":"ContainerStarted","Data":"2c93fa7a05592e1352a687f981d2bca8622191545b10ddd0e87d79e7528b5ee1"} Oct 30 00:27:05.334944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:05.329543 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-43dc72-5f4bd85fdc-2fn22"] Oct 30 00:27:05.345653 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:27:05.345617 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c04baf6_1ee8_47ee_b3a7_3f92393b644b.slice/crio-b47c117f61967e5f75dfa05f4c849af88091f2a2ecb24f0aca6f54ed6f01a384 WatchSource:0}: Error finding container b47c117f61967e5f75dfa05f4c849af88091f2a2ecb24f0aca6f54ed6f01a384: Status 404 returned error can't find the container with id b47c117f61967e5f75dfa05f4c849af88091f2a2ecb24f0aca6f54ed6f01a384 Oct 30 00:27:05.428014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:05.416304 5001 patch_prober.go:29] interesting pod/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 00:27:05.428014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:05.416364 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" podUID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 00:27:06.299850 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.298731 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-43dc72-5f4bd85fdc-2fn22" event={"ID":"6c04baf6-1ee8-47ee-b3a7-3f92393b644b","Type":"ContainerStarted","Data":"61ff95047d6a666808973edaa21c3a1550c167ac068dc6a34f148966f060b52a"} Oct 30 00:27:06.299850 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.298782 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-43dc72-5f4bd85fdc-2fn22" event={"ID":"6c04baf6-1ee8-47ee-b3a7-3f92393b644b","Type":"ContainerStarted","Data":"b47c117f61967e5f75dfa05f4c849af88091f2a2ecb24f0aca6f54ed6f01a384"} Oct 30 00:27:06.299850 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.299800 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-43dc72-5f4bd85fdc-2fn22" Oct 30 00:27:06.314429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.303113 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" event={"ID":"a684c9be-f1fe-420b-a3a0-16ec60772131","Type":"ContainerStarted","Data":"06ebf0b626bf6de28c6b1d380ef384c11f214f1de1bab6ea1cac5ce013619781"} Oct 30 00:27:06.333982 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.333946 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk"] Oct 30 00:27:06.339401 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.339380 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:27:06.355943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.355113 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-20d942-openshift-default-dockercfg-8lwt2\"" Oct 30 00:27:06.371394 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.370452 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-43dc72-5f4bd85fdc-2fn22" podStartSLOduration=2.370439997 podStartE2EDuration="2.370439997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:27:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:27:06.338402619 +0000 UTC m=+1341.543903822" watchObservedRunningTime="2026-10-30 00:27:06.370439997 +0000 UTC m=+1341.575941178" Oct 30 00:27:06.381943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.380868 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk"] Oct 30 00:27:06.386232 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.385927 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" podStartSLOduration=3.385899986 podStartE2EDuration="3.385899986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:27:06.382854112 +0000 UTC m=+1341.588355294" watchObservedRunningTime="2026-10-30 00:27:06.385899986 +0000 UTC m=+1341.591401234" Oct 30 00:27:06.416354 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.416322 5001 patch_prober.go:29] interesting pod/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 00:27:06.416999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.416963 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" podUID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 00:27:06.456072 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.456038 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91eb024d-d850-4682-9821-bcae4cb40b49-istiod-ca-cert\") pod \"e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:27:06.456263 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.456180 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-istio-envoy\") pod \"e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:27:06.456356 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.456332 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-istio-data\") pod \"e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:27:06.456475 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.456458 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-workload-certs\") pod \"e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:27:06.456615 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.456592 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91eb024d-d850-4682-9821-bcae4cb40b49-istio-podinfo\") pod \"e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:27:06.456727 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.456712 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91eb024d-d850-4682-9821-bcae4cb40b49-istio-token\") pod \"e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:27:06.456765 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.456741 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7phqh\" (UniqueName: \"kubernetes.io/projected/91eb024d-d850-4682-9821-bcae4cb40b49-kube-api-access-7phqh\") pod \"e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:27:06.456765 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.456761 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-workload-socket\") pod \"e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:27:06.456859 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.456842 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-credential-socket\") pod \"e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:27:06.588009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.581088 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91eb024d-d850-4682-9821-bcae4cb40b49-istio-podinfo\") pod \"e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:27:06.588009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.581182 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91eb024d-d850-4682-9821-bcae4cb40b49-istio-token\") pod \"e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:27:06.588009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.581207 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7phqh\" (UniqueName: \"kubernetes.io/projected/91eb024d-d850-4682-9821-bcae4cb40b49-kube-api-access-7phqh\") pod \"e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:27:06.588009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.581238 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-workload-socket\") pod \"e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:27:06.588009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.581307 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-credential-socket\") pod \"e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:27:06.588009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.581427 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91eb024d-d850-4682-9821-bcae4cb40b49-istiod-ca-cert\") pod \"e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:27:06.588009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.581552 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-istio-envoy\") pod \"e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:27:06.588009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.581615 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-istio-data\") pod \"e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:27:06.588009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.581714 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-workload-certs\") pod \"e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:27:06.588009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.582311 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-workload-socket\") pod \"e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:27:06.588009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.583475 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91eb024d-d850-4682-9821-bcae4cb40b49-istiod-ca-cert\") pod \"e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:27:06.588009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.583732 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-credential-socket\") pod \"e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:27:06.588009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.584807 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-istio-data\") pod \"e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:27:06.613945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.613530 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-istio-envoy\") pod \"e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:27:06.617940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.617040 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-workload-certs\") pod \"e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:27:06.627738 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.627702 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91eb024d-d850-4682-9821-bcae4cb40b49-istio-podinfo\") pod \"e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:27:06.630177 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.630118 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91eb024d-d850-4682-9821-bcae4cb40b49-istio-token\") pod \"e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:27:06.638961 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.637304 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7phqh\" (UniqueName: \"kubernetes.io/projected/91eb024d-d850-4682-9821-bcae4cb40b49-kube-api-access-7phqh\") pod \"e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:27:06.675564 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:06.675293 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:27:07.135444 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:07.135413 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk"] Oct 30 00:27:07.136122 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:27:07.136093 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91eb024d_d850_4682_9821_bcae4cb40b49.slice/crio-680e4b2678ce03bd561c30306a579164c8d54518a6c9479162613a80d25eba2b WatchSource:0}: Error finding container 680e4b2678ce03bd561c30306a579164c8d54518a6c9479162613a80d25eba2b: Status 404 returned error can't find the container with id 680e4b2678ce03bd561c30306a579164c8d54518a6c9479162613a80d25eba2b Oct 30 00:27:07.139925 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:07.139871 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:27:07.140023 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:07.139983 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:27:07.140069 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:07.140027 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:27:07.290198 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:07.290157 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:27:07.309281 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:07.309243 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:27:07.333606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:07.333571 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" event={"ID":"91eb024d-d850-4682-9821-bcae4cb40b49","Type":"ContainerStarted","Data":"680e4b2678ce03bd561c30306a579164c8d54518a6c9479162613a80d25eba2b"} Oct 30 00:27:07.334076 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:07.334047 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:27:07.335132 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:07.335099 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:27:07.422668 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:07.420355 5001 patch_prober.go:29] interesting pod/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 00:27:07.422668 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:07.420414 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" podUID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 00:27:07.681300 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:07.681257 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:27:07.681403 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:07.681361 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:27:07.681451 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:07.681439 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:27:08.346427 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:08.346391 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" event={"ID":"91eb024d-d850-4682-9821-bcae4cb40b49","Type":"ContainerStarted","Data":"8eacc60f7c5c21d0a438dc1a6137c99db7dddde626d7aa5a33f5807edbde3bfa"} Oct 30 00:27:08.348611 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:08.348583 5001 generic.go:356] "Generic (PLEG): container finished" podID="809701ad-afba-418f-8844-36e0baa399df" containerID="05dded4bfe99293e12b3924e62f99c1ecd6e4749594d2d1d2a5be65e83171360" exitCode=0 Oct 30 00:27:08.348768 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:08.348651 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" event={"ID":"809701ad-afba-418f-8844-36e0baa399df","Type":"ContainerDied","Data":"05dded4bfe99293e12b3924e62f99c1ecd6e4749594d2d1d2a5be65e83171360"} Oct 30 00:27:08.348768 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:08.348695 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" event={"ID":"809701ad-afba-418f-8844-36e0baa399df","Type":"ContainerStarted","Data":"bc3f1408c65e20151b5bc9f87e6baa20216ca6c4a1e183ec92d2074897819519"} Oct 30 00:27:08.396646 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:08.396586 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" podStartSLOduration=2.39655568 podStartE2EDuration="2.39655568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:27:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:27:08.39495328 +0000 UTC m=+1343.600454464" watchObservedRunningTime="2026-10-30 00:27:08.39655568 +0000 UTC m=+1343.602056862" Oct 30 00:27:08.416334 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:08.416300 5001 patch_prober.go:29] interesting pod/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 00:27:08.416524 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:08.416359 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" podUID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 00:27:08.562637 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:08.562605 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:27:08.563055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:08.562958 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:27:08.563342 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:08.563222 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:27:08.687985 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:08.685186 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:27:08.712429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:08.712005 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:27:08.767333 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:08.767300 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw"] Oct 30 00:27:08.767588 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:08.767554 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" podUID="3f2a45ad-8a47-428c-b70f-515b50d98abd" containerName="istio-proxy" containerID="cri-o://0fa1b578cb799bd148c2b287a38e31c32ae1d1d52c05a865c6e789893bc378ee" gracePeriod=30 Oct 30 00:27:09.382476 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:09.382445 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:27:09.385934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:09.384095 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:27:09.416967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:09.416679 5001 patch_prober.go:29] interesting pod/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 00:27:09.416967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:09.416744 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" podUID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 00:27:09.559750 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:09.559718 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:27:09.559942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:09.559781 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:27:10.416885 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:10.416856 5001 patch_prober.go:29] interesting pod/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 00:27:10.417317 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:10.416905 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" podUID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 00:27:10.559756 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:10.559715 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:27:10.559959 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:10.559783 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:27:11.416509 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:11.416474 5001 patch_prober.go:29] interesting pod/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 00:27:11.416692 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:11.416530 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" podUID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 00:27:11.559198 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:11.559165 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:27:11.559584 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:11.559218 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:27:11.816289 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:11.816251 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd"] Oct 30 00:27:11.821748 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:11.821721 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:27:11.834439 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:11.834394 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-4500d5-openshift-default-dockercfg-vg6gr\"" Oct 30 00:27:11.853781 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:11.853747 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd"] Oct 30 00:27:11.899668 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:11.899629 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-istio-data\") pod \"e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:27:11.899849 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:11.899794 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/242dd865-71a7-4ade-a9ae-e80adc01cf59-istio-token\") pod \"e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:27:11.899849 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:11.899840 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/242dd865-71a7-4ade-a9ae-e80adc01cf59-istio-podinfo\") pod \"e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:27:11.900015 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:11.899990 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-workload-socket\") pod \"e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:27:11.900093 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:11.900074 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/242dd865-71a7-4ade-a9ae-e80adc01cf59-istiod-ca-cert\") pod \"e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:27:11.900149 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:11.900132 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-credential-socket\") pod \"e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:27:11.900225 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:11.900202 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j669\" (UniqueName: \"kubernetes.io/projected/242dd865-71a7-4ade-a9ae-e80adc01cf59-kube-api-access-8j669\") pod \"e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:27:11.900286 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:11.900264 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-istio-envoy\") pod \"e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:27:11.900380 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:11.900366 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-workload-certs\") pod \"e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:27:12.013231 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:12.013155 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-istio-data\") pod \"e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:27:12.013421 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:12.013332 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/242dd865-71a7-4ade-a9ae-e80adc01cf59-istio-token\") pod \"e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:27:12.013421 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:12.013401 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/242dd865-71a7-4ade-a9ae-e80adc01cf59-istio-podinfo\") pod \"e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:27:12.013550 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:12.013499 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-workload-socket\") pod \"e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:27:12.016934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:12.013627 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/242dd865-71a7-4ade-a9ae-e80adc01cf59-istiod-ca-cert\") pod \"e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:27:12.016934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:12.013664 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-credential-socket\") pod \"e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:27:12.016934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:12.013742 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j669\" (UniqueName: \"kubernetes.io/projected/242dd865-71a7-4ade-a9ae-e80adc01cf59-kube-api-access-8j669\") pod \"e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:27:12.016934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:12.013864 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-istio-envoy\") pod \"e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:27:12.016934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:12.013963 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-workload-certs\") pod \"e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:27:12.023772 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:12.023736 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/242dd865-71a7-4ade-a9ae-e80adc01cf59-istiod-ca-cert\") pod \"e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:27:12.024303 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:12.024280 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-istio-data\") pod \"e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:27:12.049934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:12.026393 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-credential-socket\") pod \"e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:27:12.049934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:12.030802 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-workload-certs\") pod \"e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:27:12.049934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:12.038565 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-workload-socket\") pod \"e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:27:12.101982 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:12.082718 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-istio-envoy\") pod \"e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:27:12.101982 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:12.085339 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j669\" (UniqueName: \"kubernetes.io/projected/242dd865-71a7-4ade-a9ae-e80adc01cf59-kube-api-access-8j669\") pod \"e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:27:12.116048 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:12.114433 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/242dd865-71a7-4ade-a9ae-e80adc01cf59-istio-token\") pod \"e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:27:12.117054 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:12.117024 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/242dd865-71a7-4ade-a9ae-e80adc01cf59-istio-podinfo\") pod \"e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:27:12.147952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:12.145219 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:27:12.429992 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:12.428172 5001 patch_prober.go:29] interesting pod/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 00:27:12.429992 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:12.428252 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" podUID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 00:27:12.571228 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:12.571168 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:27:12.571931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:12.571883 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:27:12.830932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:12.821004 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-20d942-9ddb98f76-2bg4x"] Oct 30 00:27:12.986734 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:12.936076 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-20d942-9ddb98f76-2bg4x" Oct 30 00:27:13.051947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:13.049763 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-20d942\"" Oct 30 00:27:13.077934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:13.064501 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b18b95b-c9bb-4019-a6fe-79435057127c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-20d942-9ddb98f76-2bg4x\" (UID: \"1b18b95b-c9bb-4019-a6fe-79435057127c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-20d942-9ddb98f76-2bg4x" Oct 30 00:27:13.077934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:13.064576 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7vbn\" (UniqueName: \"kubernetes.io/projected/1b18b95b-c9bb-4019-a6fe-79435057127c-kube-api-access-g7vbn\") pod \"payload-processing-e2e-praxis-b-20d942-9ddb98f76-2bg4x\" (UID: \"1b18b95b-c9bb-4019-a6fe-79435057127c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-20d942-9ddb98f76-2bg4x" Oct 30 00:27:13.077934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:13.075144 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-20d942-dockercfg-d2kg9\"" Oct 30 00:27:13.249990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:13.249182 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b18b95b-c9bb-4019-a6fe-79435057127c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-20d942-9ddb98f76-2bg4x\" (UID: \"1b18b95b-c9bb-4019-a6fe-79435057127c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-20d942-9ddb98f76-2bg4x" Oct 30 00:27:13.249990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:13.249267 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7vbn\" (UniqueName: \"kubernetes.io/projected/1b18b95b-c9bb-4019-a6fe-79435057127c-kube-api-access-g7vbn\") pod \"payload-processing-e2e-praxis-b-20d942-9ddb98f76-2bg4x\" (UID: \"1b18b95b-c9bb-4019-a6fe-79435057127c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-20d942-9ddb98f76-2bg4x" Oct 30 00:27:13.254174 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:13.251052 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b18b95b-c9bb-4019-a6fe-79435057127c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-20d942-9ddb98f76-2bg4x\" (UID: \"1b18b95b-c9bb-4019-a6fe-79435057127c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-20d942-9ddb98f76-2bg4x" Oct 30 00:27:13.371382 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:13.370783 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7vbn\" (UniqueName: \"kubernetes.io/projected/1b18b95b-c9bb-4019-a6fe-79435057127c-kube-api-access-g7vbn\") pod \"payload-processing-e2e-praxis-b-20d942-9ddb98f76-2bg4x\" (UID: \"1b18b95b-c9bb-4019-a6fe-79435057127c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-20d942-9ddb98f76-2bg4x" Oct 30 00:27:13.373664 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:13.373627 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-20d942-9ddb98f76-2bg4x"] Oct 30 00:27:13.407762 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:13.403317 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:13.435980 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:13.417417 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:27:13.468053 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:13.455955 5001 patch_prober.go:29] interesting pod/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 00:27:13.468053 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:13.456013 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" podUID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 00:27:13.505609 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:13.500780 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-20d942-9ddb98f76-2bg4x" Oct 30 00:27:13.602136 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:13.582063 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:13.602136 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:13.584816 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd"] Oct 30 00:27:13.602136 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:13.586120 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:27:13.602136 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:13.586169 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:27:13.602136 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:13.600683 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:27:13.602136 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:13.600772 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:27:13.602136 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:13.600818 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:27:13.761221 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:13.761186 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" event={"ID":"242dd865-71a7-4ade-a9ae-e80adc01cf59","Type":"ContainerStarted","Data":"3ca7cb7aa17eff4d5d03a35e8762ef1fc12a47cab61561c39e84758377dee015"} Oct 30 00:27:13.761385 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:13.761247 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" event={"ID":"242dd865-71a7-4ade-a9ae-e80adc01cf59","Type":"ContainerStarted","Data":"e02f88653545461aae9d2445c0adf04fe876b9c88547a064b7152c551351fa69"} Oct 30 00:27:13.803564 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:13.803500 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" podStartSLOduration=2.8034751829999998 podStartE2EDuration="2.803475183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:27:13.801309167 +0000 UTC m=+1349.006810343" watchObservedRunningTime="2026-10-30 00:27:13.803475183 +0000 UTC m=+1349.008976425" Oct 30 00:27:13.906502 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:27:13.906470 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b18b95b_c9bb_4019_a6fe_79435057127c.slice/crio-d272ebf7271cb9c82332d49c36c28404ab5e6ff83cf3f4fc2e7a36bc9ea7ddda WatchSource:0}: Error finding container d272ebf7271cb9c82332d49c36c28404ab5e6ff83cf3f4fc2e7a36bc9ea7ddda: Status 404 returned error can't find the container with id d272ebf7271cb9c82332d49c36c28404ab5e6ff83cf3f4fc2e7a36bc9ea7ddda Oct 30 00:27:13.906502 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:13.906481 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-20d942-9ddb98f76-2bg4x"] Oct 30 00:27:14.153037 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.151207 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:27:14.153037 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.152588 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 00:27:14.153037 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.152635 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" podUID="242dd865-71a7-4ade-a9ae-e80adc01cf59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 00:27:14.287139 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.287073 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:27:14.416839 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.416807 5001 patch_prober.go:29] interesting pod/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 00:27:14.417006 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.416858 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" podUID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 00:27:14.438404 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.438373 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-credential-socket\") pod \"3f2a45ad-8a47-428c-b70f-515b50d98abd\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " Oct 30 00:27:14.438600 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.438443 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f2a45ad-8a47-428c-b70f-515b50d98abd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f2a45ad-8a47-428c-b70f-515b50d98abd-istio-podinfo\") pod \"3f2a45ad-8a47-428c-b70f-515b50d98abd\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " Oct 30 00:27:14.438600 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.438554 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-workload-certs\") pod \"3f2a45ad-8a47-428c-b70f-515b50d98abd\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " Oct 30 00:27:14.438730 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.438625 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f2a45ad-8a47-428c-b70f-515b50d98abd-kube-api-access-4684w\" (UniqueName: \"kubernetes.io/projected/3f2a45ad-8a47-428c-b70f-515b50d98abd-kube-api-access-4684w\") pod \"3f2a45ad-8a47-428c-b70f-515b50d98abd\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " Oct 30 00:27:14.438780 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.438729 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f2a45ad-8a47-428c-b70f-515b50d98abd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f2a45ad-8a47-428c-b70f-515b50d98abd-istiod-ca-cert\") pod \"3f2a45ad-8a47-428c-b70f-515b50d98abd\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " Oct 30 00:27:14.438863 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.438788 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-credential-socket" pod "3f2a45ad-8a47-428c-b70f-515b50d98abd" (UID: "3f2a45ad-8a47-428c-b70f-515b50d98abd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:14.438984 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.438862 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-istio-data\") pod \"3f2a45ad-8a47-428c-b70f-515b50d98abd\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " Oct 30 00:27:14.459045 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.438986 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f2a45ad-8a47-428c-b70f-515b50d98abd-istio-token\" (UniqueName: \"kubernetes.io/projected/3f2a45ad-8a47-428c-b70f-515b50d98abd-istio-token\") pod \"3f2a45ad-8a47-428c-b70f-515b50d98abd\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " Oct 30 00:27:14.459045 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.439039 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-istio-envoy\") pod \"3f2a45ad-8a47-428c-b70f-515b50d98abd\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " Oct 30 00:27:14.459045 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.439125 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-workload-socket\") pod \"3f2a45ad-8a47-428c-b70f-515b50d98abd\" (UID: \"3f2a45ad-8a47-428c-b70f-515b50d98abd\") " Oct 30 00:27:14.459045 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.439207 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f2a45ad-8a47-428c-b70f-515b50d98abd-istiod-ca-cert" pod "3f2a45ad-8a47-428c-b70f-515b50d98abd" (UID: "3f2a45ad-8a47-428c-b70f-515b50d98abd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:27:14.459045 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.439807 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-workload-certs" pod "3f2a45ad-8a47-428c-b70f-515b50d98abd" (UID: "3f2a45ad-8a47-428c-b70f-515b50d98abd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:14.459045 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.439891 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-istio-data" pod "3f2a45ad-8a47-428c-b70f-515b50d98abd" (UID: "3f2a45ad-8a47-428c-b70f-515b50d98abd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:14.459045 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.440114 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:14.459045 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.440161 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:14.459045 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.440176 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f2a45ad-8a47-428c-b70f-515b50d98abd-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:14.459045 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.440190 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:14.459045 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.443991 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-workload-socket" pod "3f2a45ad-8a47-428c-b70f-515b50d98abd" (UID: "3f2a45ad-8a47-428c-b70f-515b50d98abd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:14.460341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.460283 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f2a45ad-8a47-428c-b70f-515b50d98abd-kube-api-access-4684w" pod "3f2a45ad-8a47-428c-b70f-515b50d98abd" (UID: "3f2a45ad-8a47-428c-b70f-515b50d98abd"). InnerVolumeSpecName "kube-api-access-4684w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:27:14.474791 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.472796 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-istio-envoy" pod "3f2a45ad-8a47-428c-b70f-515b50d98abd" (UID: "3f2a45ad-8a47-428c-b70f-515b50d98abd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:14.474791 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.472948 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f2a45ad-8a47-428c-b70f-515b50d98abd-istio-podinfo" pod "3f2a45ad-8a47-428c-b70f-515b50d98abd" (UID: "3f2a45ad-8a47-428c-b70f-515b50d98abd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:27:14.476506 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.476442 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f2a45ad-8a47-428c-b70f-515b50d98abd-istio-token" pod "3f2a45ad-8a47-428c-b70f-515b50d98abd" (UID: "3f2a45ad-8a47-428c-b70f-515b50d98abd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:27:14.558501 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.558419 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f2a45ad-8a47-428c-b70f-515b50d98abd-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:14.558730 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.558710 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:14.565930 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.558844 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2a45ad-8a47-428c-b70f-515b50d98abd-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:14.566249 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.566229 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f2a45ad-8a47-428c-b70f-515b50d98abd-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:14.566375 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.566360 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4684w\" (UniqueName: \"kubernetes.io/projected/3f2a45ad-8a47-428c-b70f-515b50d98abd-kube-api-access-4684w\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:14.566453 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.566188 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:27:14.566587 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.566563 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:27:14.772274 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.772240 5001 generic.go:356] "Generic (PLEG): container finished" podID="3f2a45ad-8a47-428c-b70f-515b50d98abd" containerID="0fa1b578cb799bd148c2b287a38e31c32ae1d1d52c05a865c6e789893bc378ee" exitCode=0 Oct 30 00:27:14.772749 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.772287 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" Oct 30 00:27:14.772749 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.772321 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" event={"ID":"3f2a45ad-8a47-428c-b70f-515b50d98abd","Type":"ContainerDied","Data":"0fa1b578cb799bd148c2b287a38e31c32ae1d1d52c05a865c6e789893bc378ee"} Oct 30 00:27:14.772749 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.772405 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw" event={"ID":"3f2a45ad-8a47-428c-b70f-515b50d98abd","Type":"ContainerDied","Data":"515a21fc198f975b9478ba852b415564cdf78db03f54205833fd6853af2cf433"} Oct 30 00:27:14.772749 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.772435 5001 scope.go:122] "RemoveContainer" containerID="0fa1b578cb799bd148c2b287a38e31c32ae1d1d52c05a865c6e789893bc378ee" Oct 30 00:27:14.774485 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.774458 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-20d942-9ddb98f76-2bg4x" event={"ID":"1b18b95b-c9bb-4019-a6fe-79435057127c","Type":"ContainerStarted","Data":"ead4bab7f69df7e23c858d749c09fd53700719c76a65b7372740f702496df36c"} Oct 30 00:27:14.774605 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.774517 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-20d942-9ddb98f76-2bg4x" event={"ID":"1b18b95b-c9bb-4019-a6fe-79435057127c","Type":"ContainerStarted","Data":"d272ebf7271cb9c82332d49c36c28404ab5e6ff83cf3f4fc2e7a36bc9ea7ddda"} Oct 30 00:27:14.774605 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.774548 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-20d942-9ddb98f76-2bg4x" Oct 30 00:27:14.784650 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.784634 5001 scope.go:122] "RemoveContainer" containerID="0fa1b578cb799bd148c2b287a38e31c32ae1d1d52c05a865c6e789893bc378ee" Oct 30 00:27:14.784976 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:27:14.784960 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fa1b578cb799bd148c2b287a38e31c32ae1d1d52c05a865c6e789893bc378ee\": container with ID starting with 0fa1b578cb799bd148c2b287a38e31c32ae1d1d52c05a865c6e789893bc378ee not found: ID does not exist" containerID="0fa1b578cb799bd148c2b287a38e31c32ae1d1d52c05a865c6e789893bc378ee" Oct 30 00:27:14.785052 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.784988 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fa1b578cb799bd148c2b287a38e31c32ae1d1d52c05a865c6e789893bc378ee"} err="failed to get container status \"0fa1b578cb799bd148c2b287a38e31c32ae1d1d52c05a865c6e789893bc378ee\": rpc error: code = NotFound desc = could not find container \"0fa1b578cb799bd148c2b287a38e31c32ae1d1d52c05a865c6e789893bc378ee\": container with ID starting with 0fa1b578cb799bd148c2b287a38e31c32ae1d1d52c05a865c6e789893bc378ee not found: ID does not exist" Oct 30 00:27:14.820411 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.820331 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw"] Oct 30 00:27:14.829660 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.829631 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-hnblw"] Oct 30 00:27:14.864653 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.864596 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-20d942-9ddb98f76-2bg4x" podStartSLOduration=2.864582308 podStartE2EDuration="2.864582308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:27:14.858729279 +0000 UTC m=+1350.064230459" watchObservedRunningTime="2026-10-30 00:27:14.864582308 +0000 UTC m=+1350.070083488" Oct 30 00:27:14.921922 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:14.921883 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f2a45ad-8a47-428c-b70f-515b50d98abd" path="/var/lib/kubelet/pods/3f2a45ad-8a47-428c-b70f-515b50d98abd/volumes" Oct 30 00:27:15.146995 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:15.146891 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 00:27:15.147250 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:15.147220 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" podUID="242dd865-71a7-4ade-a9ae-e80adc01cf59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 00:27:15.416425 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:15.416337 5001 patch_prober.go:29] interesting pod/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 00:27:15.416425 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:15.416402 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" podUID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 00:27:15.559293 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:15.559260 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:27:15.559502 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:15.559481 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:27:15.559548 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:15.559525 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:27:16.149396 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:16.149359 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 00:27:16.149936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:16.149455 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" podUID="242dd865-71a7-4ade-a9ae-e80adc01cf59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 00:27:16.417172 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:16.417085 5001 patch_prober.go:29] interesting pod/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 00:27:16.417172 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:16.417142 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" podUID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 00:27:16.559852 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:16.559814 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:27:16.560060 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:16.559881 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:27:17.146297 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:17.146243 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 00:27:17.146498 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:17.146304 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" podUID="242dd865-71a7-4ade-a9ae-e80adc01cf59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 00:27:17.357934 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:27:17.357600 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:27:17.422632 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:17.422559 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:27:17.423704 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:17.423684 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:27:17.564426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:17.564395 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:27:17.565562 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:17.565541 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:27:18.310387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:18.310353 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:27:18.341819 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:18.312261 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:27:18.341819 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:18.341738 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:27:18.374431 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:18.374139 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-43dc72-5f4bd85fdc-2fn22" Oct 30 00:27:21.971517 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:21.971464 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 30 00:27:22.127119 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:22.127070 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:22.237988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:22.227642 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:22.390081 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:22.390023 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:22.503150 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:22.503047 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:27:22.741449 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:22.741414 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2"] Oct 30 00:27:22.767248 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:22.757140 5001 cpu_manager.go:414] "containerMap: removing container" podUID="3f2a45ad-8a47-428c-b70f-515b50d98abd" containerName="istio-proxy" Oct 30 00:27:22.767248 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:22.757171 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f2a45ad-8a47-428c-b70f-515b50d98abd" containerName="istio-proxy" Oct 30 00:27:22.767248 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:22.757417 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f2a45ad-8a47-428c-b70f-515b50d98abd" containerName="istio-proxy" Oct 30 00:27:22.848620 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:22.847347 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:27:22.869964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:22.864576 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-d3e13d-openshift-default-dockercfg-6c8sg\"" Oct 30 00:27:22.938945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:22.927388 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istio-data\") pod \"e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:27:22.977133 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:22.977098 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-credential-socket\") pod \"e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:27:22.977881 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:22.977853 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istiod-ca-cert\") pod \"e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:27:22.978106 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:22.978063 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:27:22.978320 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:22.978296 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istio-podinfo\") pod \"e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:27:22.978444 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:22.978428 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istio-token\") pod \"e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:27:22.978751 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:22.978718 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bqwc\" (UniqueName: \"kubernetes.io/projected/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-kube-api-access-7bqwc\") pod \"e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:27:22.979473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:22.978903 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-workload-socket\") pod \"e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:27:23.003444 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:23.003410 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istio-envoy\") pod \"e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:27:23.003786 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:23.003761 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-workload-certs\") pod \"e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:27:23.024419 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:23.023040 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 00:27:23.026294 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:23.025281 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:23.115964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:23.106639 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istio-podinfo\") pod \"e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:27:23.115964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:23.106806 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istio-token\") pod \"e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:27:23.115964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:23.106924 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bqwc\" (UniqueName: \"kubernetes.io/projected/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-kube-api-access-7bqwc\") pod \"e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:27:23.115964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:23.106977 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-workload-socket\") pod \"e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:27:23.115964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:23.107035 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istio-envoy\") pod \"e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:27:23.115964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:23.107128 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-workload-certs\") pod \"e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:27:23.115964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:23.107185 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istio-data\") pod \"e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:27:23.115964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:23.107248 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-credential-socket\") pod \"e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:27:23.115964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:23.107615 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istiod-ca-cert\") pod \"e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:27:23.115964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:23.113427 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-workload-socket\") pod \"e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:27:23.139101 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:23.138382 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2"] Oct 30 00:27:23.139101 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:23.138486 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-workload-certs\") pod \"e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:27:23.139101 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:23.138928 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istio-data\") pod \"e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:27:23.139400 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:23.139179 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-credential-socket\") pod \"e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:27:23.157818 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:23.157710 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istiod-ca-cert\") pod \"e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:27:23.248933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:23.244677 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:23.248933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:23.245027 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:27:23.319189 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:23.319100 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istio-token\") pod \"e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:27:23.418498 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:23.341579 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istio-podinfo\") pod \"e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:27:23.418498 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:23.407407 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:27:23.418498 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:23.411521 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:23.418498 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:23.416504 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:27:23.511058 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:23.511005 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:23.554935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:23.529217 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:27:23.759989 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:23.644000 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bqwc\" (UniqueName: \"kubernetes.io/projected/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-kube-api-access-7bqwc\") pod \"e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:27:23.759989 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:23.679989 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istio-envoy\") pod \"e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:27:23.888465 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:23.888423 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:27:24.028934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:24.026652 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:24.040134 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:24.040076 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 00:27:24.053147 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:24.053102 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:27:24.098091 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:24.055129 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 30 00:27:24.098517 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:24.055160 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:27:24.098517 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:24.084534 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:24.098517 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:24.084571 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/health\": context deadline exceeded" Oct 30 00:27:24.099411 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:24.099387 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:27:24.099871 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:24.099838 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:27:24.294006 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:24.265471 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 00:27:24.436057 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:24.435995 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 00:27:24.526529 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:24.526463 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:27:24.554174 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:24.554080 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:27:24.558338 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:24.554861 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:27:25.047937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:25.029123 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:25.049939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:25.048572 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:25.103371 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:25.103316 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 30 00:27:25.118542 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:25.103644 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:27:25.170405 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:25.156821 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:27:25.343713 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:25.343064 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv"] Oct 30 00:27:25.343713 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:25.343419 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" podUID="f84427bf-e708-4616-b20d-417182380594" containerName="istio-proxy" containerID="cri-o://af3bb49da4fc6376eadeaa96b3d75d783bef98e419cbfcc5faa13d2c481ea04d" gracePeriod=30 Oct 30 00:27:25.520049 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:25.498682 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:25.668054 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:25.637002 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:27:25.754062 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:25.734160 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:27:25.755679 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:25.754480 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2"] Oct 30 00:27:25.755679 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:25.754968 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" podUID="00301bd4-1afd-4dd3-8fbc-45376008859e" containerName="istio-proxy" containerID="cri-o://95aa6314df85021ded6bef94b0eb2dcfafa4905540c3906fb1620cf13d953503" gracePeriod=30 Oct 30 00:27:25.955269 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:25.955165 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:27:26.028076 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:26.028036 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:27:26.138064 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:26.138014 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:26.210255 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:26.189556 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:26.210734 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:26.209711 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn"] Oct 30 00:27:26.213612 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:26.209865 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:26.304049 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:26.283893 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 00:27:26.373860 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:26.325615 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" podUID="8c596e5d-d36f-4543-836e-15ee5d3b111b" containerName="istio-proxy" containerID="cri-o://376c865cfccf8e75227c21f756202b3cfbed20b6950d75aa58a781a1093a6d36" gracePeriod=30 Oct 30 00:27:26.374406 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:26.362792 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-20d942-9ddb98f76-2bg4x" Oct 30 00:27:26.616942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:26.610721 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:27:26.617554 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:26.617519 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:27:26.643992 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:26.643953 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:27:26.644769 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:26.637819 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:26.708235 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:26.694239 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:27:26.757171 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:26.736304 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:27.028083 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:27.028031 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:27.256973 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:27.239534 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:27.256973 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:27.239933 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:27:27.257856 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:27.257602 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:27.319172 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:27.295554 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2"] Oct 30 00:27:27.382553 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:27.381989 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 00:27:27.420936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:27.413664 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:27:27.748023 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:27.747929 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:27:28.014174 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:28.009151 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 00:27:28.038603 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:28.038555 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 00:27:28.040489 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:28.040466 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:27:28.065452 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:28.065410 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:28.266988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:28.258723 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 30 00:27:28.266988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:28.259354 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:27:28.306721 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:28.306696 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:27:28.425324 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:28.425030 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" event={"ID":"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6","Type":"ContainerStarted","Data":"02b8bbb99fd50c202822d113a2aa0456014e60c70caaaa058a5d9e9a665e4e87"} Oct 30 00:27:28.425324 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:28.425096 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" event={"ID":"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6","Type":"ContainerStarted","Data":"c99a98fa006c35d537b5f052547f846f673e2ec42f05e76a52d2f5ee91d3c033"} Oct 30 00:27:28.500967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:28.500167 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": context deadline exceeded" Oct 30 00:27:28.600885 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:28.598412 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:27:28.883583 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:28.883491 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:27:28.972936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:28.890039 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:27:29.013575 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:29.013532 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:29.348943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:29.339017 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 00:27:29.471262 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:29.471216 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:29.518237 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:29.518192 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:29.669767 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:29.655751 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:27:29.787935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:29.785097 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:29.914782 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:29.914747 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:27:29.920246 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:29.920158 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:27:29.923646 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:29.923592 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" podStartSLOduration=7.75990675 podStartE2EDuration="7.75990675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:27:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:27:28.539568164 +0000 UTC m=+1363.745069344" watchObservedRunningTime="2026-10-30 00:27:29.75990675 +0000 UTC m=+1364.965407932" Oct 30 00:27:29.954941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:29.948503 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:27:30.245006 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:30.239303 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:30.342034 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:30.341987 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 30 00:27:30.497993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:30.480301 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:27:30.497993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:30.480709 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:30.551581 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:30.525115 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:27:30.663059 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:30.659395 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:30.663059 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:30.660105 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:27:30.818001 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:30.817967 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:27:30.819004 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:30.818972 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:27:30.820174 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:30.820153 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:27:30.904222 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:30.895403 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:27:31.676321 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:31.675826 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:27:31.715954 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:31.681385 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 00:27:31.718406 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:31.716347 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:31.833807 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:31.833760 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:31.834655 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:31.834622 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:27:31.881089 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:31.880262 5001 generic.go:356] "Generic (PLEG): container finished" podID="f84427bf-e708-4616-b20d-417182380594" containerID="af3bb49da4fc6376eadeaa96b3d75d783bef98e419cbfcc5faa13d2c481ea04d" exitCode=0 Oct 30 00:27:31.881627 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:31.880395 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" event={"ID":"f84427bf-e708-4616-b20d-417182380594","Type":"ContainerDied","Data":"af3bb49da4fc6376eadeaa96b3d75d783bef98e419cbfcc5faa13d2c481ea04d"} Oct 30 00:27:31.883257 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:31.883143 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:27:31.968890 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:31.968781 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:32.005935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:32.003305 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:27:32.114016 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:32.113971 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:32.303751 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:32.297054 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 00:27:32.468455 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:32.467628 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:32.679108 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:32.679001 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:32.719301 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:32.696149 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:32.719301 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:32.716690 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:32.840379 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:32.839268 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:33.043130 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:32.982754 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:33.335382 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:33.323081 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:33.481290 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:33.481232 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 30 00:27:33.564937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:33.540805 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:27:33.643608 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:33.641038 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:27:33.643608 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:33.641078 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" event={"ID":"8c596e5d-d36f-4543-836e-15ee5d3b111b","Type":"ContainerDied","Data":"376c865cfccf8e75227c21f756202b3cfbed20b6950d75aa58a781a1093a6d36"} Oct 30 00:27:33.643608 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:33.641129 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp"] Oct 30 00:27:33.643608 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:33.641201 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:27:33.643608 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:33.641279 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m"] Oct 30 00:27:33.643608 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:33.641330 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:27:33.643608 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:33.641365 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:27:33.643608 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:33.641382 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7"] Oct 30 00:27:33.643608 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:33.641574 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" podUID="374db7c3-3cfd-4abf-9199-27f55fd2c548" containerName="istio-proxy" containerID="cri-o://72658ef052b0c90bdf5341bf06233b2b465936fd985225664e1c22b713934b41" gracePeriod=30 Oct 30 00:27:33.643608 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:33.605131 5001 generic.go:356] "Generic (PLEG): container finished" podID="8c596e5d-d36f-4543-836e-15ee5d3b111b" containerID="376c865cfccf8e75227c21f756202b3cfbed20b6950d75aa58a781a1093a6d36" exitCode=0 Oct 30 00:27:33.645850 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:33.644348 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" podUID="38fbc17b-0fb6-46c8-83af-53b83fa2bcb1" containerName="istio-proxy" containerID="cri-o://add4d613b8644e6634338b28efdc601d2fa4cad732d30520b5434a4cdd351bbe" gracePeriod=30 Oct 30 00:27:33.645850 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:33.644585 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" podUID="5e44a9bb-130f-4749-bd51-ab0fa7958247" containerName="istio-proxy" containerID="cri-o://aeb62f4e111c193e872f476b470f7547ff79ed11912b31bea78ac5f3dc4e9a91" gracePeriod=30 Oct 30 00:27:33.879843 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:33.832155 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:33.937683 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:33.881158 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:33.937683 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:33.899929 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:27:33.959434 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:33.958193 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/health\": context deadline exceeded" Oct 30 00:27:34.012677 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:34.012639 5001 generic.go:356] "Generic (PLEG): container finished" podID="00301bd4-1afd-4dd3-8fbc-45376008859e" containerID="95aa6314df85021ded6bef94b0eb2dcfafa4905540c3906fb1620cf13d953503" exitCode=0 Oct 30 00:27:34.013113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:34.013087 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" event={"ID":"00301bd4-1afd-4dd3-8fbc-45376008859e","Type":"ContainerDied","Data":"95aa6314df85021ded6bef94b0eb2dcfafa4905540c3906fb1620cf13d953503"} Oct 30 00:27:34.099940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:34.099867 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:34.325932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:34.324184 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:34.511567 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:34.511521 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:34.634130 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:34.634046 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:27:34.641828 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:34.641798 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:27:34.876935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:34.838509 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 00:27:34.999294 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:34.999167 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:35.504266 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.504233 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:27:35.542844 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.504988 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:27:35.581524 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.562502 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:27:35.581833 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.581582 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:27:35.595097 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.591789 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:27:35.595097 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.591823 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" event={"ID":"8c596e5d-d36f-4543-836e-15ee5d3b111b","Type":"ContainerDied","Data":"961e9b45c8dc7a670c8f644bffd674705a23cadb968d76da7b1d2e2743f41d7e"} Oct 30 00:27:35.595097 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.591856 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="961e9b45c8dc7a670c8f644bffd674705a23cadb968d76da7b1d2e2743f41d7e" Oct 30 00:27:35.596463 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.596432 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" event={"ID":"f84427bf-e708-4616-b20d-417182380594","Type":"ContainerDied","Data":"853fb4480acfeb024a06839974c6034d5d420c36b860d8363f8409f207c58a5b"} Oct 30 00:27:35.596667 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.596649 5001 scope.go:122] "RemoveContainer" containerID="af3bb49da4fc6376eadeaa96b3d75d783bef98e419cbfcc5faa13d2c481ea04d" Oct 30 00:27:35.599677 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.599599 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" event={"ID":"00301bd4-1afd-4dd3-8fbc-45376008859e","Type":"ContainerDied","Data":"a238a5bfd17338d00e992b770a7777903378a0eb12e8fb2cb5b1a5876ca03481"} Oct 30 00:27:35.621124 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.621081 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2" Oct 30 00:27:35.621482 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.621462 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:27:35.623819 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.623792 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:27:35.624016 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.623991 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" podUID="f84427bf-e708-4616-b20d-417182380594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:35.640947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.639131 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-workload-certs\") pod \"00301bd4-1afd-4dd3-8fbc-45376008859e\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " Oct 30 00:27:35.640947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.639195 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00301bd4-1afd-4dd3-8fbc-45376008859e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00301bd4-1afd-4dd3-8fbc-45376008859e-istiod-ca-cert\") pod \"00301bd4-1afd-4dd3-8fbc-45376008859e\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " Oct 30 00:27:35.640947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.639231 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00301bd4-1afd-4dd3-8fbc-45376008859e-istio-token\" (UniqueName: \"kubernetes.io/projected/00301bd4-1afd-4dd3-8fbc-45376008859e-istio-token\") pod \"00301bd4-1afd-4dd3-8fbc-45376008859e\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " Oct 30 00:27:35.640947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.639269 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-istio-envoy\") pod \"00301bd4-1afd-4dd3-8fbc-45376008859e\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " Oct 30 00:27:35.640947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.639317 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-credential-socket\") pod \"00301bd4-1afd-4dd3-8fbc-45376008859e\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " Oct 30 00:27:35.640947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.639371 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00301bd4-1afd-4dd3-8fbc-45376008859e-kube-api-access-nxxqs\" (UniqueName: \"kubernetes.io/projected/00301bd4-1afd-4dd3-8fbc-45376008859e-kube-api-access-nxxqs\") pod \"00301bd4-1afd-4dd3-8fbc-45376008859e\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " Oct 30 00:27:35.640947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.639412 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f84427bf-e708-4616-b20d-417182380594-kube-api-access-l67x5\" (UniqueName: \"kubernetes.io/projected/f84427bf-e708-4616-b20d-417182380594-kube-api-access-l67x5\") pod \"f84427bf-e708-4616-b20d-417182380594\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " Oct 30 00:27:35.640947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.639589 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f84427bf-e708-4616-b20d-417182380594-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f84427bf-e708-4616-b20d-417182380594-istio-podinfo\") pod \"f84427bf-e708-4616-b20d-417182380594\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " Oct 30 00:27:35.640947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.639659 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-credential-socket\") pod \"f84427bf-e708-4616-b20d-417182380594\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " Oct 30 00:27:35.640947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.639692 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-workload-certs\") pod \"f84427bf-e708-4616-b20d-417182380594\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " Oct 30 00:27:35.640947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.639752 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-istio-data\") pod \"00301bd4-1afd-4dd3-8fbc-45376008859e\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " Oct 30 00:27:35.640947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.639798 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f84427bf-e708-4616-b20d-417182380594-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f84427bf-e708-4616-b20d-417182380594-istiod-ca-cert\") pod \"f84427bf-e708-4616-b20d-417182380594\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " Oct 30 00:27:35.640947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.639858 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-workload-socket\") pod \"00301bd4-1afd-4dd3-8fbc-45376008859e\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " Oct 30 00:27:35.640947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.639980 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00301bd4-1afd-4dd3-8fbc-45376008859e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00301bd4-1afd-4dd3-8fbc-45376008859e-istio-podinfo\") pod \"00301bd4-1afd-4dd3-8fbc-45376008859e\" (UID: \"00301bd4-1afd-4dd3-8fbc-45376008859e\") " Oct 30 00:27:35.640947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.640020 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-istio-envoy\") pod \"f84427bf-e708-4616-b20d-417182380594\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " Oct 30 00:27:35.640947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.640132 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f84427bf-e708-4616-b20d-417182380594-istio-token\" (UniqueName: \"kubernetes.io/projected/f84427bf-e708-4616-b20d-417182380594-istio-token\") pod \"f84427bf-e708-4616-b20d-417182380594\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " Oct 30 00:27:35.640947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.640178 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-istio-data\") pod \"f84427bf-e708-4616-b20d-417182380594\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " Oct 30 00:27:35.640947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.640220 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-workload-socket\") pod \"f84427bf-e708-4616-b20d-417182380594\" (UID: \"f84427bf-e708-4616-b20d-417182380594\") " Oct 30 00:27:35.643490 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.643315 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-credential-socket" pod "f84427bf-e708-4616-b20d-417182380594" (UID: "f84427bf-e708-4616-b20d-417182380594"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:35.643890 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.643678 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-istio-data" pod "00301bd4-1afd-4dd3-8fbc-45376008859e" (UID: "00301bd4-1afd-4dd3-8fbc-45376008859e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:35.644233 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.644046 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-workload-certs" pod "00301bd4-1afd-4dd3-8fbc-45376008859e" (UID: "00301bd4-1afd-4dd3-8fbc-45376008859e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:35.644816 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.644626 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-workload-certs" pod "f84427bf-e708-4616-b20d-417182380594" (UID: "f84427bf-e708-4616-b20d-417182380594"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:35.645408 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.645127 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00301bd4-1afd-4dd3-8fbc-45376008859e-istiod-ca-cert" pod "00301bd4-1afd-4dd3-8fbc-45376008859e" (UID: "00301bd4-1afd-4dd3-8fbc-45376008859e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:27:35.645817 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.645545 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-workload-socket" pod "f84427bf-e708-4616-b20d-417182380594" (UID: "f84427bf-e708-4616-b20d-417182380594"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:35.658261 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.658029 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00301bd4-1afd-4dd3-8fbc-45376008859e-istio-podinfo" pod "00301bd4-1afd-4dd3-8fbc-45376008859e" (UID: "00301bd4-1afd-4dd3-8fbc-45376008859e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:27:35.658850 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.658586 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84427bf-e708-4616-b20d-417182380594-istiod-ca-cert" pod "f84427bf-e708-4616-b20d-417182380594" (UID: "f84427bf-e708-4616-b20d-417182380594"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:27:35.660165 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.659033 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-workload-socket" pod "00301bd4-1afd-4dd3-8fbc-45376008859e" (UID: "00301bd4-1afd-4dd3-8fbc-45376008859e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:35.665442 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.665386 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f84427bf-e708-4616-b20d-417182380594-kube-api-access-l67x5" pod "f84427bf-e708-4616-b20d-417182380594" (UID: "f84427bf-e708-4616-b20d-417182380594"). InnerVolumeSpecName "kube-api-access-l67x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:27:35.666579 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.666072 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00301bd4-1afd-4dd3-8fbc-45376008859e-kube-api-access-nxxqs" pod "00301bd4-1afd-4dd3-8fbc-45376008859e" (UID: "00301bd4-1afd-4dd3-8fbc-45376008859e"). InnerVolumeSpecName "kube-api-access-nxxqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:27:35.666579 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.666331 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-credential-socket" pod "00301bd4-1afd-4dd3-8fbc-45376008859e" (UID: "00301bd4-1afd-4dd3-8fbc-45376008859e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:35.668177 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.668149 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-istio-data" pod "f84427bf-e708-4616-b20d-417182380594" (UID: "f84427bf-e708-4616-b20d-417182380594"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:35.670882 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.668727 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:35.670882 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.668753 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:35.670882 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.668768 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:35.670882 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.668782 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00301bd4-1afd-4dd3-8fbc-45376008859e-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:35.670882 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.668796 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:35.670882 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.668809 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxxqs\" (UniqueName: \"kubernetes.io/projected/00301bd4-1afd-4dd3-8fbc-45376008859e-kube-api-access-nxxqs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:35.670882 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.668823 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l67x5\" (UniqueName: \"kubernetes.io/projected/f84427bf-e708-4616-b20d-417182380594-kube-api-access-l67x5\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:35.670882 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.668837 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:35.670882 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.668851 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:35.670882 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.668864 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:35.670882 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.668878 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f84427bf-e708-4616-b20d-417182380594-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:35.670882 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.668892 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:35.672010 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.671947 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00301bd4-1afd-4dd3-8fbc-45376008859e-istio-token" pod "00301bd4-1afd-4dd3-8fbc-45376008859e" (UID: "00301bd4-1afd-4dd3-8fbc-45376008859e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:27:35.672610 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.672566 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00301bd4-1afd-4dd3-8fbc-45376008859e-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:35.674942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.674896 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f84427bf-e708-4616-b20d-417182380594-istio-podinfo" pod "f84427bf-e708-4616-b20d-417182380594" (UID: "f84427bf-e708-4616-b20d-417182380594"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:27:35.675231 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.675199 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f84427bf-e708-4616-b20d-417182380594-istio-token" pod "f84427bf-e708-4616-b20d-417182380594" (UID: "f84427bf-e708-4616-b20d-417182380594"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:27:35.679950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.679772 5001 scope.go:122] "RemoveContainer" containerID="95aa6314df85021ded6bef94b0eb2dcfafa4905540c3906fb1620cf13d953503" Oct 30 00:27:35.683750 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.683707 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-istio-envoy" pod "00301bd4-1afd-4dd3-8fbc-45376008859e" (UID: "00301bd4-1afd-4dd3-8fbc-45376008859e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:35.701178 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.701129 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-istio-envoy" pod "f84427bf-e708-4616-b20d-417182380594" (UID: "f84427bf-e708-4616-b20d-417182380594"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:35.779996 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.778857 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-workload-socket\") pod \"8c596e5d-d36f-4543-836e-15ee5d3b111b\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " Oct 30 00:27:35.779996 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.778961 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-credential-socket\") pod \"8c596e5d-d36f-4543-836e-15ee5d3b111b\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " Oct 30 00:27:35.779996 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.779055 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c596e5d-d36f-4543-836e-15ee5d3b111b-kube-api-access-fntfg\" (UniqueName: \"kubernetes.io/projected/8c596e5d-d36f-4543-836e-15ee5d3b111b-kube-api-access-fntfg\") pod \"8c596e5d-d36f-4543-836e-15ee5d3b111b\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " Oct 30 00:27:35.779996 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.779097 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-workload-certs\") pod \"8c596e5d-d36f-4543-836e-15ee5d3b111b\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " Oct 30 00:27:35.779996 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.779179 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c596e5d-d36f-4543-836e-15ee5d3b111b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c596e5d-d36f-4543-836e-15ee5d3b111b-istio-podinfo\") pod \"8c596e5d-d36f-4543-836e-15ee5d3b111b\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " Oct 30 00:27:35.779996 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.779256 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-istio-data\") pod \"8c596e5d-d36f-4543-836e-15ee5d3b111b\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " Oct 30 00:27:35.779996 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.779402 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c596e5d-d36f-4543-836e-15ee5d3b111b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c596e5d-d36f-4543-836e-15ee5d3b111b-istiod-ca-cert\") pod \"8c596e5d-d36f-4543-836e-15ee5d3b111b\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " Oct 30 00:27:35.779996 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.779445 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c596e5d-d36f-4543-836e-15ee5d3b111b-istio-token\" (UniqueName: \"kubernetes.io/projected/8c596e5d-d36f-4543-836e-15ee5d3b111b-istio-token\") pod \"8c596e5d-d36f-4543-836e-15ee5d3b111b\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " Oct 30 00:27:35.779996 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.779570 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-istio-envoy\") pod \"8c596e5d-d36f-4543-836e-15ee5d3b111b\" (UID: \"8c596e5d-d36f-4543-836e-15ee5d3b111b\") " Oct 30 00:27:35.783934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.781932 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f84427bf-e708-4616-b20d-417182380594-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:35.783934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.781962 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f84427bf-e708-4616-b20d-417182380594-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:35.783934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.781976 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00301bd4-1afd-4dd3-8fbc-45376008859e-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:35.783934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.781990 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00301bd4-1afd-4dd3-8fbc-45376008859e-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:35.783934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.782005 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f84427bf-e708-4616-b20d-417182380594-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:35.785442 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.785144 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c596e5d-d36f-4543-836e-15ee5d3b111b-istiod-ca-cert" pod "8c596e5d-d36f-4543-836e-15ee5d3b111b" (UID: "8c596e5d-d36f-4543-836e-15ee5d3b111b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:27:35.786555 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.786498 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-workload-socket" pod "8c596e5d-d36f-4543-836e-15ee5d3b111b" (UID: "8c596e5d-d36f-4543-836e-15ee5d3b111b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:35.786751 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.786714 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-credential-socket" pod "8c596e5d-d36f-4543-836e-15ee5d3b111b" (UID: "8c596e5d-d36f-4543-836e-15ee5d3b111b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:35.786922 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.786887 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-workload-certs" pod "8c596e5d-d36f-4543-836e-15ee5d3b111b" (UID: "8c596e5d-d36f-4543-836e-15ee5d3b111b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:35.788021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.787996 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-istio-data" pod "8c596e5d-d36f-4543-836e-15ee5d3b111b" (UID: "8c596e5d-d36f-4543-836e-15ee5d3b111b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:35.794961 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.794481 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c596e5d-d36f-4543-836e-15ee5d3b111b-kube-api-access-fntfg" pod "8c596e5d-d36f-4543-836e-15ee5d3b111b" (UID: "8c596e5d-d36f-4543-836e-15ee5d3b111b"). InnerVolumeSpecName "kube-api-access-fntfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:27:35.797882 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.797853 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-istio-envoy" pod "8c596e5d-d36f-4543-836e-15ee5d3b111b" (UID: "8c596e5d-d36f-4543-836e-15ee5d3b111b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:35.798215 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.798174 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c596e5d-d36f-4543-836e-15ee5d3b111b-istio-token" pod "8c596e5d-d36f-4543-836e-15ee5d3b111b" (UID: "8c596e5d-d36f-4543-836e-15ee5d3b111b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:27:35.798349 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.798318 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c596e5d-d36f-4543-836e-15ee5d3b111b-istio-podinfo" pod "8c596e5d-d36f-4543-836e-15ee5d3b111b" (UID: "8c596e5d-d36f-4543-836e-15ee5d3b111b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:27:35.883625 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.883470 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:35.883625 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.883504 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c596e5d-d36f-4543-836e-15ee5d3b111b-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:35.883625 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.883517 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c596e5d-d36f-4543-836e-15ee5d3b111b-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:35.883625 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.883529 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:35.883625 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.883544 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:35.883625 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.883557 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:35.883625 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.883571 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fntfg\" (UniqueName: \"kubernetes.io/projected/8c596e5d-d36f-4543-836e-15ee5d3b111b-kube-api-access-fntfg\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:35.883625 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.883584 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c596e5d-d36f-4543-836e-15ee5d3b111b-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:35.883625 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:35.883597 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c596e5d-d36f-4543-836e-15ee5d3b111b-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:36.061810 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:36.061696 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2"] Oct 30 00:27:36.070362 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:36.070321 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-hm9w2"] Oct 30 00:27:36.438516 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:36.438014 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:27:36.438516 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:36.438109 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" podUID="8c596e5d-d36f-4543-836e-15ee5d3b111b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:36.646842 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:36.646811 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv" Oct 30 00:27:36.652986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:36.649942 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn" Oct 30 00:27:36.707515 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:36.707449 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv"] Oct 30 00:27:36.882933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:36.882011 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-jprdv"] Oct 30 00:27:37.126027 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:37.122261 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00301bd4-1afd-4dd3-8fbc-45376008859e" path="/var/lib/kubelet/pods/00301bd4-1afd-4dd3-8fbc-45376008859e/volumes" Oct 30 00:27:37.126027 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:37.123150 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f84427bf-e708-4616-b20d-417182380594" path="/var/lib/kubelet/pods/f84427bf-e708-4616-b20d-417182380594/volumes" Oct 30 00:27:37.126027 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:37.123668 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn"] Oct 30 00:27:37.163943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:37.159733 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-kkzjn"] Oct 30 00:27:38.962802 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:38.962373 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c596e5d-d36f-4543-836e-15ee5d3b111b" path="/var/lib/kubelet/pods/8c596e5d-d36f-4543-836e-15ee5d3b111b/volumes" Oct 30 00:27:39.252001 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.250728 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:27:39.350099 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.349137 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-workload-certs\") pod \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " Oct 30 00:27:39.350099 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.349240 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istio-podinfo\") pod \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " Oct 30 00:27:39.350099 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.349278 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-workload-socket\") pod \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " Oct 30 00:27:39.350099 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.349493 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istio-envoy\") pod \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " Oct 30 00:27:39.350099 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.349539 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-kube-api-access-bpsn5\" (UniqueName: \"kubernetes.io/projected/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-kube-api-access-bpsn5\") pod \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " Oct 30 00:27:39.350099 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.349581 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istiod-ca-cert\") pod \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " Oct 30 00:27:39.350099 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.349693 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istio-data\") pod \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " Oct 30 00:27:39.350099 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.349774 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istio-token\" (UniqueName: \"kubernetes.io/projected/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istio-token\") pod \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " Oct 30 00:27:39.350099 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.349817 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-credential-socket\") pod \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\" (UID: \"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1\") " Oct 30 00:27:39.357970 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.357207 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-workload-certs" pod "38fbc17b-0fb6-46c8-83af-53b83fa2bcb1" (UID: "38fbc17b-0fb6-46c8-83af-53b83fa2bcb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:39.359402 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.358201 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istio-data" pod "38fbc17b-0fb6-46c8-83af-53b83fa2bcb1" (UID: "38fbc17b-0fb6-46c8-83af-53b83fa2bcb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:39.359402 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.358586 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istiod-ca-cert" pod "38fbc17b-0fb6-46c8-83af-53b83fa2bcb1" (UID: "38fbc17b-0fb6-46c8-83af-53b83fa2bcb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:27:39.359402 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.359360 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-credential-socket" pod "38fbc17b-0fb6-46c8-83af-53b83fa2bcb1" (UID: "38fbc17b-0fb6-46c8-83af-53b83fa2bcb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:39.360241 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.359838 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-workload-socket" pod "38fbc17b-0fb6-46c8-83af-53b83fa2bcb1" (UID: "38fbc17b-0fb6-46c8-83af-53b83fa2bcb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:39.360241 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.360134 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:39.360241 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.360160 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:39.360241 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.360176 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:39.360241 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.360191 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:39.360241 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.360207 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:39.377261 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.369028 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istio-envoy" pod "38fbc17b-0fb6-46c8-83af-53b83fa2bcb1" (UID: "38fbc17b-0fb6-46c8-83af-53b83fa2bcb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:39.377261 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.369086 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-kube-api-access-bpsn5" pod "38fbc17b-0fb6-46c8-83af-53b83fa2bcb1" (UID: "38fbc17b-0fb6-46c8-83af-53b83fa2bcb1"). InnerVolumeSpecName "kube-api-access-bpsn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:27:39.377261 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.369517 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istio-token" pod "38fbc17b-0fb6-46c8-83af-53b83fa2bcb1" (UID: "38fbc17b-0fb6-46c8-83af-53b83fa2bcb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:27:39.378659 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.377499 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istio-podinfo" pod "38fbc17b-0fb6-46c8-83af-53b83fa2bcb1" (UID: "38fbc17b-0fb6-46c8-83af-53b83fa2bcb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:27:39.466554 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.464209 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:39.466554 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.464244 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:39.466554 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.464259 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpsn5\" (UniqueName: \"kubernetes.io/projected/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-kube-api-access-bpsn5\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:39.466554 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.464274 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:39.771945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.769675 5001 generic.go:356] "Generic (PLEG): container finished" podID="38fbc17b-0fb6-46c8-83af-53b83fa2bcb1" containerID="add4d613b8644e6634338b28efdc601d2fa4cad732d30520b5434a4cdd351bbe" exitCode=0 Oct 30 00:27:39.771945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.769978 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" event={"ID":"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1","Type":"ContainerDied","Data":"add4d613b8644e6634338b28efdc601d2fa4cad732d30520b5434a4cdd351bbe"} Oct 30 00:27:39.771945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.770036 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" event={"ID":"38fbc17b-0fb6-46c8-83af-53b83fa2bcb1","Type":"ContainerDied","Data":"e694ebafde43c1e0180b70777d44d56a2bc337709658cc8b509724f068ecf20f"} Oct 30 00:27:39.771945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.770070 5001 scope.go:122] "RemoveContainer" containerID="add4d613b8644e6634338b28efdc601d2fa4cad732d30520b5434a4cdd351bbe" Oct 30 00:27:39.771945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.770228 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m" Oct 30 00:27:39.778501 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.778305 5001 generic.go:356] "Generic (PLEG): container finished" podID="5e44a9bb-130f-4749-bd51-ab0fa7958247" containerID="aeb62f4e111c193e872f476b470f7547ff79ed11912b31bea78ac5f3dc4e9a91" exitCode=0 Oct 30 00:27:39.778501 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.778387 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" event={"ID":"5e44a9bb-130f-4749-bd51-ab0fa7958247","Type":"ContainerDied","Data":"aeb62f4e111c193e872f476b470f7547ff79ed11912b31bea78ac5f3dc4e9a91"} Oct 30 00:27:39.793026 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.792992 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:27:39.857810 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.857781 5001 scope.go:122] "RemoveContainer" containerID="add4d613b8644e6634338b28efdc601d2fa4cad732d30520b5434a4cdd351bbe" Oct 30 00:27:39.873422 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:27:39.859355 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"add4d613b8644e6634338b28efdc601d2fa4cad732d30520b5434a4cdd351bbe\": container with ID starting with add4d613b8644e6634338b28efdc601d2fa4cad732d30520b5434a4cdd351bbe not found: ID does not exist" containerID="add4d613b8644e6634338b28efdc601d2fa4cad732d30520b5434a4cdd351bbe" Oct 30 00:27:39.873422 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.859420 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"add4d613b8644e6634338b28efdc601d2fa4cad732d30520b5434a4cdd351bbe"} err="failed to get container status \"add4d613b8644e6634338b28efdc601d2fa4cad732d30520b5434a4cdd351bbe\": rpc error: code = NotFound desc = could not find container \"add4d613b8644e6634338b28efdc601d2fa4cad732d30520b5434a4cdd351bbe\": container with ID starting with add4d613b8644e6634338b28efdc601d2fa4cad732d30520b5434a4cdd351bbe not found: ID does not exist" Oct 30 00:27:39.877962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.877189 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-istio-envoy\") pod \"5e44a9bb-130f-4749-bd51-ab0fa7958247\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " Oct 30 00:27:39.877962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.877250 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-credential-socket\") pod \"5e44a9bb-130f-4749-bd51-ab0fa7958247\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " Oct 30 00:27:39.877962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.877388 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e44a9bb-130f-4749-bd51-ab0fa7958247-kube-api-access-6rg5b\" (UniqueName: \"kubernetes.io/projected/5e44a9bb-130f-4749-bd51-ab0fa7958247-kube-api-access-6rg5b\") pod \"5e44a9bb-130f-4749-bd51-ab0fa7958247\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " Oct 30 00:27:39.877962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.877415 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e44a9bb-130f-4749-bd51-ab0fa7958247-istio-token\" (UniqueName: \"kubernetes.io/projected/5e44a9bb-130f-4749-bd51-ab0fa7958247-istio-token\") pod \"5e44a9bb-130f-4749-bd51-ab0fa7958247\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " Oct 30 00:27:39.877962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.877494 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-istio-data\") pod \"5e44a9bb-130f-4749-bd51-ab0fa7958247\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " Oct 30 00:27:39.877962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.877567 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e44a9bb-130f-4749-bd51-ab0fa7958247-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e44a9bb-130f-4749-bd51-ab0fa7958247-istio-podinfo\") pod \"5e44a9bb-130f-4749-bd51-ab0fa7958247\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " Oct 30 00:27:39.877962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.877600 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-workload-certs\") pod \"5e44a9bb-130f-4749-bd51-ab0fa7958247\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " Oct 30 00:27:39.877962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.877765 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-workload-socket\") pod \"5e44a9bb-130f-4749-bd51-ab0fa7958247\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " Oct 30 00:27:39.877962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.877813 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e44a9bb-130f-4749-bd51-ab0fa7958247-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e44a9bb-130f-4749-bd51-ab0fa7958247-istiod-ca-cert\") pod \"5e44a9bb-130f-4749-bd51-ab0fa7958247\" (UID: \"5e44a9bb-130f-4749-bd51-ab0fa7958247\") " Oct 30 00:27:39.880107 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.879785 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e44a9bb-130f-4749-bd51-ab0fa7958247-istiod-ca-cert" pod "5e44a9bb-130f-4749-bd51-ab0fa7958247" (UID: "5e44a9bb-130f-4749-bd51-ab0fa7958247"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:27:39.881564 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.881229 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-workload-socket" pod "5e44a9bb-130f-4749-bd51-ab0fa7958247" (UID: "5e44a9bb-130f-4749-bd51-ab0fa7958247"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:39.881564 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.881490 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-credential-socket" pod "5e44a9bb-130f-4749-bd51-ab0fa7958247" (UID: "5e44a9bb-130f-4749-bd51-ab0fa7958247"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:39.881564 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.881543 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-workload-certs" pod "5e44a9bb-130f-4749-bd51-ab0fa7958247" (UID: "5e44a9bb-130f-4749-bd51-ab0fa7958247"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:39.882027 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.882010 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-istio-data" pod "5e44a9bb-130f-4749-bd51-ab0fa7958247" (UID: "5e44a9bb-130f-4749-bd51-ab0fa7958247"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:39.897959 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.891554 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e44a9bb-130f-4749-bd51-ab0fa7958247-kube-api-access-6rg5b" pod "5e44a9bb-130f-4749-bd51-ab0fa7958247" (UID: "5e44a9bb-130f-4749-bd51-ab0fa7958247"). InnerVolumeSpecName "kube-api-access-6rg5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:27:39.897959 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.893043 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e44a9bb-130f-4749-bd51-ab0fa7958247-istio-token" pod "5e44a9bb-130f-4749-bd51-ab0fa7958247" (UID: "5e44a9bb-130f-4749-bd51-ab0fa7958247"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:27:39.914026 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.912430 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e44a9bb-130f-4749-bd51-ab0fa7958247-istio-podinfo" pod "5e44a9bb-130f-4749-bd51-ab0fa7958247" (UID: "5e44a9bb-130f-4749-bd51-ab0fa7958247"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:27:39.914026 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.913865 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-istio-envoy" pod "5e44a9bb-130f-4749-bd51-ab0fa7958247" (UID: "5e44a9bb-130f-4749-bd51-ab0fa7958247"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:39.953681 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.953588 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m"] Oct 30 00:27:39.986524 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.982947 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:39.986524 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.982997 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e44a9bb-130f-4749-bd51-ab0fa7958247-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:39.986524 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.983015 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:39.986524 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.983032 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:39.986524 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.983049 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rg5b\" (UniqueName: \"kubernetes.io/projected/5e44a9bb-130f-4749-bd51-ab0fa7958247-kube-api-access-6rg5b\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:39.986524 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.983072 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e44a9bb-130f-4749-bd51-ab0fa7958247-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:39.986524 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.983088 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:39.986524 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.983103 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e44a9bb-130f-4749-bd51-ab0fa7958247-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:39.986524 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:39.983118 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e44a9bb-130f-4749-bd51-ab0fa7958247-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:40.001090 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:40.001056 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-rs54m"] Oct 30 00:27:40.349688 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:40.349360 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:27:40.414074 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:40.412578 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-workload-certs\") pod \"374db7c3-3cfd-4abf-9199-27f55fd2c548\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " Oct 30 00:27:40.414074 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:40.412641 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/374db7c3-3cfd-4abf-9199-27f55fd2c548-kube-api-access-slfrz\" (UniqueName: \"kubernetes.io/projected/374db7c3-3cfd-4abf-9199-27f55fd2c548-kube-api-access-slfrz\") pod \"374db7c3-3cfd-4abf-9199-27f55fd2c548\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " Oct 30 00:27:40.414074 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:40.412674 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-workload-socket\") pod \"374db7c3-3cfd-4abf-9199-27f55fd2c548\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " Oct 30 00:27:40.414074 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:40.412699 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/374db7c3-3cfd-4abf-9199-27f55fd2c548-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/374db7c3-3cfd-4abf-9199-27f55fd2c548-istiod-ca-cert\") pod \"374db7c3-3cfd-4abf-9199-27f55fd2c548\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " Oct 30 00:27:40.414074 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:40.412854 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/374db7c3-3cfd-4abf-9199-27f55fd2c548-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/374db7c3-3cfd-4abf-9199-27f55fd2c548-istio-podinfo\") pod \"374db7c3-3cfd-4abf-9199-27f55fd2c548\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " Oct 30 00:27:40.414074 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:40.412928 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-istio-data\") pod \"374db7c3-3cfd-4abf-9199-27f55fd2c548\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " Oct 30 00:27:40.414074 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:40.412991 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-credential-socket\") pod \"374db7c3-3cfd-4abf-9199-27f55fd2c548\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " Oct 30 00:27:40.414074 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:40.413055 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/374db7c3-3cfd-4abf-9199-27f55fd2c548-istio-token\" (UniqueName: \"kubernetes.io/projected/374db7c3-3cfd-4abf-9199-27f55fd2c548-istio-token\") pod \"374db7c3-3cfd-4abf-9199-27f55fd2c548\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " Oct 30 00:27:40.414074 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:40.413140 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-istio-envoy\") pod \"374db7c3-3cfd-4abf-9199-27f55fd2c548\" (UID: \"374db7c3-3cfd-4abf-9199-27f55fd2c548\") " Oct 30 00:27:40.414545 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:40.414096 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/374db7c3-3cfd-4abf-9199-27f55fd2c548-istiod-ca-cert" pod "374db7c3-3cfd-4abf-9199-27f55fd2c548" (UID: "374db7c3-3cfd-4abf-9199-27f55fd2c548"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:27:40.414988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:40.414802 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-credential-socket" pod "374db7c3-3cfd-4abf-9199-27f55fd2c548" (UID: "374db7c3-3cfd-4abf-9199-27f55fd2c548"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:40.414988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:40.414947 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-istio-data" pod "374db7c3-3cfd-4abf-9199-27f55fd2c548" (UID: "374db7c3-3cfd-4abf-9199-27f55fd2c548"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:40.416173 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:40.416095 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-workload-socket" pod "374db7c3-3cfd-4abf-9199-27f55fd2c548" (UID: "374db7c3-3cfd-4abf-9199-27f55fd2c548"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:40.416369 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:40.416337 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-workload-certs" pod "374db7c3-3cfd-4abf-9199-27f55fd2c548" (UID: "374db7c3-3cfd-4abf-9199-27f55fd2c548"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:40.419947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:40.419885 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-istio-envoy" pod "374db7c3-3cfd-4abf-9199-27f55fd2c548" (UID: "374db7c3-3cfd-4abf-9199-27f55fd2c548"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:27:40.423185 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:40.423146 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/374db7c3-3cfd-4abf-9199-27f55fd2c548-kube-api-access-slfrz" pod "374db7c3-3cfd-4abf-9199-27f55fd2c548" (UID: "374db7c3-3cfd-4abf-9199-27f55fd2c548"). InnerVolumeSpecName "kube-api-access-slfrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:27:40.432371 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:40.432335 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/374db7c3-3cfd-4abf-9199-27f55fd2c548-istio-token" pod "374db7c3-3cfd-4abf-9199-27f55fd2c548" (UID: "374db7c3-3cfd-4abf-9199-27f55fd2c548"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:27:40.437780 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:40.437743 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/374db7c3-3cfd-4abf-9199-27f55fd2c548-istio-podinfo" pod "374db7c3-3cfd-4abf-9199-27f55fd2c548" (UID: "374db7c3-3cfd-4abf-9199-27f55fd2c548"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:27:40.516017 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:40.515839 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:40.516017 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:40.515877 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slfrz\" (UniqueName: \"kubernetes.io/projected/374db7c3-3cfd-4abf-9199-27f55fd2c548-kube-api-access-slfrz\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:40.516017 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:40.515894 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:40.516017 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:40.515924 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/374db7c3-3cfd-4abf-9199-27f55fd2c548-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:40.516017 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:40.515939 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/374db7c3-3cfd-4abf-9199-27f55fd2c548-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:40.516017 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:40.515953 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:40.516017 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:40.515968 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:40.516017 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:40.515981 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/374db7c3-3cfd-4abf-9199-27f55fd2c548-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:40.516017 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:40.515994 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/374db7c3-3cfd-4abf-9199-27f55fd2c548-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:27:41.022331 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:41.021634 5001 generic.go:356] "Generic (PLEG): container finished" podID="374db7c3-3cfd-4abf-9199-27f55fd2c548" containerID="72658ef052b0c90bdf5341bf06233b2b465936fd985225664e1c22b713934b41" exitCode=0 Oct 30 00:27:41.022331 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:41.021763 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" Oct 30 00:27:41.070816 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:41.069589 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" Oct 30 00:27:41.091038 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:41.089768 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38fbc17b-0fb6-46c8-83af-53b83fa2bcb1" path="/var/lib/kubelet/pods/38fbc17b-0fb6-46c8-83af-53b83fa2bcb1/volumes" Oct 30 00:27:41.091038 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:41.090839 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" event={"ID":"374db7c3-3cfd-4abf-9199-27f55fd2c548","Type":"ContainerDied","Data":"72658ef052b0c90bdf5341bf06233b2b465936fd985225664e1c22b713934b41"} Oct 30 00:27:41.091038 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:41.090863 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7" event={"ID":"374db7c3-3cfd-4abf-9199-27f55fd2c548","Type":"ContainerDied","Data":"eb491fcd2da1b40e3a8c26cebf9d42805cefc1524ac0362113becf8a2f63d640"} Oct 30 00:27:41.091038 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:41.090876 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp" event={"ID":"5e44a9bb-130f-4749-bd51-ab0fa7958247","Type":"ContainerDied","Data":"b3854becd30e59666f815f15a5d435219bf7cec36b218df41ecf39cdce0c6096"} Oct 30 00:27:41.091038 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:41.090901 5001 scope.go:122] "RemoveContainer" containerID="72658ef052b0c90bdf5341bf06233b2b465936fd985225664e1c22b713934b41" Oct 30 00:27:41.137492 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:41.137465 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7"] Oct 30 00:27:41.188231 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:41.188165 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-dcjx7"] Oct 30 00:27:41.197823 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:41.197753 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp"] Oct 30 00:27:41.205785 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:41.205741 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wrzpp"] Oct 30 00:27:41.217995 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:41.217372 5001 scope.go:122] "RemoveContainer" containerID="72658ef052b0c90bdf5341bf06233b2b465936fd985225664e1c22b713934b41" Oct 30 00:27:41.218450 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:27:41.218420 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72658ef052b0c90bdf5341bf06233b2b465936fd985225664e1c22b713934b41\": container with ID starting with 72658ef052b0c90bdf5341bf06233b2b465936fd985225664e1c22b713934b41 not found: ID does not exist" containerID="72658ef052b0c90bdf5341bf06233b2b465936fd985225664e1c22b713934b41" Oct 30 00:27:41.218621 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:41.218592 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72658ef052b0c90bdf5341bf06233b2b465936fd985225664e1c22b713934b41"} err="failed to get container status \"72658ef052b0c90bdf5341bf06233b2b465936fd985225664e1c22b713934b41\": rpc error: code = NotFound desc = could not find container \"72658ef052b0c90bdf5341bf06233b2b465936fd985225664e1c22b713934b41\": container with ID starting with 72658ef052b0c90bdf5341bf06233b2b465936fd985225664e1c22b713934b41 not found: ID does not exist" Oct 30 00:27:41.218714 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:41.218623 5001 scope.go:122] "RemoveContainer" containerID="aeb62f4e111c193e872f476b470f7547ff79ed11912b31bea78ac5f3dc4e9a91" Oct 30 00:27:42.930879 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:42.930842 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="374db7c3-3cfd-4abf-9199-27f55fd2c548" path="/var/lib/kubelet/pods/374db7c3-3cfd-4abf-9199-27f55fd2c548/volumes" Oct 30 00:27:42.931636 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:42.931609 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e44a9bb-130f-4749-bd51-ab0fa7958247" path="/var/lib/kubelet/pods/5e44a9bb-130f-4749-bd51-ab0fa7958247/volumes" Oct 30 00:27:43.059964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:43.056001 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:27:43.059964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:43.056065 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:43.230304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:43.230070 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:27:43.230304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:43.230180 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" podUID="34da2ec4-9938-4eaf-9845-dc23bcbc761a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:43.684055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:43.683503 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:27:43.684055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:43.683573 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" podUID="4fb15e40-3233-4329-917c-6bfcf33cb894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:43.781258 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:43.781072 5001 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:27:43.781258 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:43.781155 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:47.199787 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:47.199755 5001 generic.go:356] "Generic (PLEG): container finished" podID="34da2ec4-9938-4eaf-9845-dc23bcbc761a" containerID="74bd4f05adabe0d4f743035fa48fd38389be4628b8c007ae54ce96696cec5737" exitCode=0 Oct 30 00:27:47.216149 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:47.199836 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" event={"ID":"34da2ec4-9938-4eaf-9845-dc23bcbc761a","Type":"ContainerDied","Data":"74bd4f05adabe0d4f743035fa48fd38389be4628b8c007ae54ce96696cec5737"} Oct 30 00:27:47.216149 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:47.200408 5001 scope.go:122] "RemoveContainer" containerID="74bd4f05adabe0d4f743035fa48fd38389be4628b8c007ae54ce96696cec5737" Oct 30 00:27:47.216149 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:47.202397 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:27:47.216149 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:47.202462 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:27:47.216149 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:47.202500 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:27:47.352956 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:27:47.352411 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:27:48.238047 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:48.238010 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" event={"ID":"34da2ec4-9938-4eaf-9845-dc23bcbc761a","Type":"ContainerStarted","Data":"f10fe7ca539096970b05937017c10fc1e492f75a6174b548cc05fd9cfd2c712b"} Oct 30 00:27:48.241137 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:48.241105 5001 generic.go:356] "Generic (PLEG): container finished" podID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerID="6171c8650a197691769b8e0b1b0674f8651df0f15863762f9b90d8d9ec49c156" exitCode=0 Oct 30 00:27:48.241252 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:48.241172 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" event={"ID":"e34c0d4a-0def-4964-8c2a-4599941a3a03","Type":"ContainerDied","Data":"6171c8650a197691769b8e0b1b0674f8651df0f15863762f9b90d8d9ec49c156"} Oct 30 00:27:48.241637 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:48.241618 5001 scope.go:122] "RemoveContainer" containerID="6171c8650a197691769b8e0b1b0674f8651df0f15863762f9b90d8d9ec49c156" Oct 30 00:27:48.243437 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:48.243418 5001 generic.go:356] "Generic (PLEG): container finished" podID="4fb15e40-3233-4329-917c-6bfcf33cb894" containerID="72ce3749930f3cca3cbe89433546c1d3a3949e2181da0067d680dad946f357b7" exitCode=0 Oct 30 00:27:48.243544 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:48.243493 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" event={"ID":"4fb15e40-3233-4329-917c-6bfcf33cb894","Type":"ContainerDied","Data":"72ce3749930f3cca3cbe89433546c1d3a3949e2181da0067d680dad946f357b7"} Oct 30 00:27:48.243804 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:48.243769 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:27:48.243881 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:48.243848 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:27:48.243971 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:48.243883 5001 scope.go:122] "RemoveContainer" containerID="72ce3749930f3cca3cbe89433546c1d3a3949e2181da0067d680dad946f357b7" Oct 30 00:27:48.243971 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:48.243889 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:27:48.245821 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:48.245793 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:27:48.245948 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:48.245869 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:27:48.245948 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:48.245935 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:27:48.246544 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:48.246524 5001 generic.go:356] "Generic (PLEG): container finished" podID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerID="eee8c4d26b92ea429508ab9169296ac584476f24181e6e3c79ae36f85e5a2b3d" exitCode=0 Oct 30 00:27:48.246637 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:48.246558 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" event={"ID":"0c35f644-80ac-439f-9efd-631b26b20a0c","Type":"ContainerDied","Data":"eee8c4d26b92ea429508ab9169296ac584476f24181e6e3c79ae36f85e5a2b3d"} Oct 30 00:27:48.247035 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:48.247016 5001 scope.go:122] "RemoveContainer" containerID="eee8c4d26b92ea429508ab9169296ac584476f24181e6e3c79ae36f85e5a2b3d" Oct 30 00:27:48.249059 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:48.249027 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:27:48.249158 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:48.249111 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:27:48.249220 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:48.249161 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:27:48.649994 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:48.619089 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:27:48.649994 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:48.619151 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:49.182237 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:49.182206 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:27:49.182453 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:49.182327 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 00:27:49.182453 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:49.182372 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" podUID="34da2ec4-9938-4eaf-9845-dc23bcbc761a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 00:27:49.335937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:49.335095 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" event={"ID":"0c35f644-80ac-439f-9efd-631b26b20a0c","Type":"ContainerStarted","Data":"7f7d43ea1bbf80ac1fa635633f532414349e8e36c67521bf3696d72bf7b9a9f7"} Oct 30 00:27:49.352933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:49.351219 5001 generic.go:356] "Generic (PLEG): container finished" podID="809701ad-afba-418f-8844-36e0baa399df" containerID="bc3f1408c65e20151b5bc9f87e6baa20216ca6c4a1e183ec92d2074897819519" exitCode=0 Oct 30 00:27:49.352933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:49.351521 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" event={"ID":"809701ad-afba-418f-8844-36e0baa399df","Type":"ContainerDied","Data":"bc3f1408c65e20151b5bc9f87e6baa20216ca6c4a1e183ec92d2074897819519"} Oct 30 00:27:49.352933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:49.351586 5001 scope.go:122] "RemoveContainer" containerID="05dded4bfe99293e12b3924e62f99c1ecd6e4749594d2d1d2a5be65e83171360" Oct 30 00:27:49.352933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:49.352294 5001 scope.go:122] "RemoveContainer" containerID="bc3f1408c65e20151b5bc9f87e6baa20216ca6c4a1e183ec92d2074897819519" Oct 30 00:27:49.377935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:49.367527 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" event={"ID":"e34c0d4a-0def-4964-8c2a-4599941a3a03","Type":"ContainerStarted","Data":"cfd0797bc44839ccf8a875a3aeb6b7c395345c4fc65808485ff08bfd7c7b00d7"} Oct 30 00:27:49.389838 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:49.386966 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" event={"ID":"4fb15e40-3233-4329-917c-6bfcf33cb894","Type":"ContainerStarted","Data":"6d6b3c1b010268cd670524f9ab127b3501e621b4281a1f0a033aac64660208e5"} Oct 30 00:27:49.417991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:49.391003 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:27:49.417991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:49.391213 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:27:49.417991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:49.391322 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:27:49.746992 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:49.725988 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:27:50.055936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:50.054543 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:27:50.055936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:50.055109 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 00:27:50.055936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:50.055164 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 00:27:50.058441 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:50.058411 5001 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 00:27:50.070134 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:50.070094 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 00:27:50.087494 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:50.087459 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:27:50.133322 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:50.108113 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 00:27:50.133322 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:50.133023 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" podUID="4fb15e40-3233-4329-917c-6bfcf33cb894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 00:27:50.208941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:50.205698 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 00:27:50.208941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:50.205761 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" podUID="34da2ec4-9938-4eaf-9845-dc23bcbc761a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 00:27:50.519962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:50.515007 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:50.735985 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:50.735039 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:50.737082 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:50.736810 5001 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 00:27:50.737082 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:50.737017 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 00:27:50.855961 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:50.833491 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 00:27:50.857305 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:50.856336 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:27:50.925585 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:50.915057 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:27:51.002029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:50.995450 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 00:27:51.002029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:50.995511 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 00:27:51.095200 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:51.067415 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:51.135940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:51.134021 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:51.203072 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:51.184714 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 00:27:51.203072 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:51.184777 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" podUID="4fb15e40-3233-4329-917c-6bfcf33cb894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 00:27:51.246250 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:51.246211 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 00:27:51.246548 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:51.246514 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" podUID="34da2ec4-9938-4eaf-9845-dc23bcbc761a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 00:27:51.436024 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:51.425001 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:51.520325 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:51.518551 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:51.520325 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:51.520237 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:27:51.555787 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:51.542553 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" event={"ID":"809701ad-afba-418f-8844-36e0baa399df","Type":"ContainerStarted","Data":"6bdbe2e5223a5aa49ca43894b9b8c419233841bc1b50dad4504dbf4c16c696ff"} Oct 30 00:27:51.562130 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:51.562099 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:27:51.562441 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:51.562419 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:27:51.562637 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:51.562609 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:27:51.738056 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:51.737962 5001 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 00:27:51.745039 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:51.744994 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 00:27:51.756944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:51.745568 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:27:51.764944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:51.758507 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:27:51.834766 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:51.834115 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:51.835241 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:51.835218 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:27:51.882681 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:51.855035 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:51.882681 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:51.864780 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:51.882681 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:51.878534 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:27:52.003214 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:52.003117 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 00:27:52.003522 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:52.003483 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 00:27:52.130989 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:52.130941 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:52.135544 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:52.135507 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:52.136237 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:52.136214 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:27:52.138803 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:52.138780 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 00:27:52.138988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:52.138963 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" podUID="4fb15e40-3233-4329-917c-6bfcf33cb894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 00:27:52.143675 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:52.143650 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 00:27:52.144166 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:52.144149 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:27:52.221874 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:52.221833 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 00:27:52.222184 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:52.222152 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" podUID="34da2ec4-9938-4eaf-9845-dc23bcbc761a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 00:27:52.471934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:52.452076 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 00:27:52.471934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:52.452467 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:27:52.549976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:52.544136 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 30 00:27:52.611065 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:52.611027 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:27:52.611385 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:52.611352 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:27:52.773949 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:52.773175 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:52.794940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:52.781140 5001 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 00:27:52.794940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:52.781203 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 00:27:52.794940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:52.794568 5001 scope.go:122] "RemoveContainer" containerID="cd530465f37e4e538e7e49642ac86280adad3148c3a1f677e070274f9dce9dd1" Oct 30 00:27:52.825874 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:52.825847 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:27:52.837289 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:52.837250 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:52.881060 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:52.880997 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:53.036021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:53.020498 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 00:27:53.036021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:53.020553 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 00:27:53.088075 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:53.088040 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 00:27:53.088323 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:53.088296 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" podUID="4fb15e40-3233-4329-917c-6bfcf33cb894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 00:27:53.139080 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:53.139038 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:53.207144 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:53.185655 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 00:27:53.207144 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:53.185711 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" podUID="34da2ec4-9938-4eaf-9845-dc23bcbc761a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 00:27:53.269937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:53.269044 5001 scope.go:122] "RemoveContainer" containerID="376c865cfccf8e75227c21f756202b3cfbed20b6950d75aa58a781a1093a6d36" Oct 30 00:27:53.455359 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:53.455319 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:53.487956 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:53.457067 5001 scope.go:122] "RemoveContainer" containerID="1681f03cd503a94f2fc185b2297d30038fd2873f8250b65a6bae1b624779f9d9" Oct 30 00:27:53.546991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:53.546017 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:53.565109 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:53.564721 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:27:53.565109 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:53.564782 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:27:53.599083 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:53.598928 5001 scope.go:122] "RemoveContainer" containerID="6e92dd1e84a9c387c52b35687018c638b294542e77c8e7e7c8fb2fc7bbf852ec" Oct 30 00:27:53.727060 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:53.727025 5001 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 00:27:53.727329 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:53.727302 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 00:27:53.755863 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:53.755808 5001 scope.go:122] "RemoveContainer" containerID="11b88c33130d0f88e2888e6f2cc6277c1b5736c4c7cc9c6d4c003fde7d5537f1" Oct 30 00:27:53.782936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:53.779007 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:27:53.872552 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:53.835013 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:53.872552 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:53.839986 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:54.005418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:54.004184 5001 scope.go:122] "RemoveContainer" containerID="c27e5730599c7cd23bffab855204bad71774a5b6f68d86462fb3a76b35cf5290" Oct 30 00:27:54.005418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:54.004398 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:54.036972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:54.009058 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:54.037437 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:54.037394 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:54.089695 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:54.053664 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 00:27:54.090612 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:54.090568 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 00:27:54.092743 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:54.083766 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 00:27:54.092992 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:54.092961 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" podUID="4fb15e40-3233-4329-917c-6bfcf33cb894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 00:27:54.207445 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:54.152532 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:54.208309 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:54.181678 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 00:27:54.219714 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:54.219632 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" podUID="34da2ec4-9938-4eaf-9845-dc23bcbc761a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 00:27:54.363375 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:54.363342 5001 scope.go:122] "RemoveContainer" containerID="b393c3711c124d85b3c1d35c1df3dc2141b77d8c1ed57574b4ead8663a3734e7" Oct 30 00:27:54.479256 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:54.457106 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:54.607328 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:54.597667 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:27:54.607328 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:54.597731 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:27:54.607328 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:54.606151 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:54.732991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:54.719889 5001 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 00:27:54.732991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:54.720206 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 00:27:54.807942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:54.802101 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:54.835403 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:54.835359 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 00:27:54.836241 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:54.836216 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:27:54.843676 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:54.843644 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:55.026055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.009563 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:55.026055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.010021 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:55.026055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.024214 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 00:27:55.026055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.024265 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 00:27:55.206952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.191528 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 00:27:55.206952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.191594 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" podUID="34da2ec4-9938-4eaf-9845-dc23bcbc761a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 00:27:55.206952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.191969 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 00:27:55.206952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.192008 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" podUID="4fb15e40-3233-4329-917c-6bfcf33cb894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 00:27:55.217160 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.217107 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:55.424936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.399553 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:27:55.534189 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.527180 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:55.670001 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.612277 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:55.670001 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.638581 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:27:55.670001 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.638635 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:27:55.765937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.762716 5001 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 00:27:55.765937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.762774 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 00:27:55.835403 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.835367 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-75dd767db7-8fdnd"] Oct 30 00:27:55.836759 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.836724 5001 cpu_manager.go:414] "containerMap: removing container" podUID="00301bd4-1afd-4dd3-8fbc-45376008859e" containerName="istio-proxy" Oct 30 00:27:55.836962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.836947 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00301bd4-1afd-4dd3-8fbc-45376008859e" containerName="istio-proxy" Oct 30 00:27:55.837090 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.837079 5001 cpu_manager.go:414] "containerMap: removing container" podUID="38fbc17b-0fb6-46c8-83af-53b83fa2bcb1" containerName="istio-proxy" Oct 30 00:27:55.837174 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.837163 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38fbc17b-0fb6-46c8-83af-53b83fa2bcb1" containerName="istio-proxy" Oct 30 00:27:55.837275 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.837265 5001 cpu_manager.go:414] "containerMap: removing container" podUID="374db7c3-3cfd-4abf-9199-27f55fd2c548" containerName="istio-proxy" Oct 30 00:27:55.837358 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.837347 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="374db7c3-3cfd-4abf-9199-27f55fd2c548" containerName="istio-proxy" Oct 30 00:27:55.837478 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.837465 5001 cpu_manager.go:414] "containerMap: removing container" podUID="5e44a9bb-130f-4749-bd51-ab0fa7958247" containerName="istio-proxy" Oct 30 00:27:55.837574 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.837563 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e44a9bb-130f-4749-bd51-ab0fa7958247" containerName="istio-proxy" Oct 30 00:27:55.837661 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.837652 5001 cpu_manager.go:414] "containerMap: removing container" podUID="8c596e5d-d36f-4543-836e-15ee5d3b111b" containerName="istio-proxy" Oct 30 00:27:55.837732 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.837722 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c596e5d-d36f-4543-836e-15ee5d3b111b" containerName="istio-proxy" Oct 30 00:27:55.837823 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.837813 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f84427bf-e708-4616-b20d-417182380594" containerName="istio-proxy" Oct 30 00:27:55.837934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.837922 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f84427bf-e708-4616-b20d-417182380594" containerName="istio-proxy" Oct 30 00:27:55.838241 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.838226 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="374db7c3-3cfd-4abf-9199-27f55fd2c548" containerName="istio-proxy" Oct 30 00:27:55.838339 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.838329 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e44a9bb-130f-4749-bd51-ab0fa7958247" containerName="istio-proxy" Oct 30 00:27:55.838568 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.838418 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c596e5d-d36f-4543-836e-15ee5d3b111b" containerName="istio-proxy" Oct 30 00:27:55.838683 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.838669 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f84427bf-e708-4616-b20d-417182380594" containerName="istio-proxy" Oct 30 00:27:55.838775 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.838765 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="00301bd4-1afd-4dd3-8fbc-45376008859e" containerName="istio-proxy" Oct 30 00:27:55.838857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.838848 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="38fbc17b-0fb6-46c8-83af-53b83fa2bcb1" containerName="istio-proxy" Oct 30 00:27:55.859698 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.859660 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:55.893037 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.866376 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-75dd767db7-8fdnd" Oct 30 00:27:55.893037 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.867564 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:55.893037 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.889855 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:27:55.969819 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.969245 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f649b0-dockercfg-gfmhg\"" Oct 30 00:27:55.970260 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.970220 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f649b0\"" Oct 30 00:27:55.984843 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:55.984799 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvlm4\" (UniqueName: \"kubernetes.io/projected/820af093-87be-4078-9098-1997a629c2c7-kube-api-access-dvlm4\") pod \"payload-pre-processing-e2e-ipp-b-f649b0-75dd767db7-8fdnd\" (UID: \"820af093-87be-4078-9098-1997a629c2c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-75dd767db7-8fdnd" Oct 30 00:27:56.101944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:56.078324 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/820af093-87be-4078-9098-1997a629c2c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f649b0-75dd767db7-8fdnd\" (UID: \"820af093-87be-4078-9098-1997a629c2c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-75dd767db7-8fdnd" Oct 30 00:27:56.101944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:56.013333 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:27:56.101944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:56.013301 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:27:56.102349 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:56.013667 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 00:27:56.244247 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:56.201992 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 00:27:56.244247 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:56.216517 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/820af093-87be-4078-9098-1997a629c2c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f649b0-75dd767db7-8fdnd\" (UID: \"820af093-87be-4078-9098-1997a629c2c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-75dd767db7-8fdnd" Oct 30 00:27:56.244247 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:56.223534 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvlm4\" (UniqueName: \"kubernetes.io/projected/820af093-87be-4078-9098-1997a629c2c7-kube-api-access-dvlm4\") pod \"payload-pre-processing-e2e-ipp-b-f649b0-75dd767db7-8fdnd\" (UID: \"820af093-87be-4078-9098-1997a629c2c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-75dd767db7-8fdnd" Oct 30 00:27:56.263989 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:56.246440 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/820af093-87be-4078-9098-1997a629c2c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f649b0-75dd767db7-8fdnd\" (UID: \"820af093-87be-4078-9098-1997a629c2c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-75dd767db7-8fdnd" Oct 30 00:27:56.275988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:56.267683 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:56.363569 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:56.342195 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 00:27:56.363569 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:56.342255 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" podUID="4fb15e40-3233-4329-917c-6bfcf33cb894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 00:27:56.363569 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:56.342692 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 00:27:56.363569 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:56.342730 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" podUID="34da2ec4-9938-4eaf-9845-dc23bcbc761a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 00:27:56.437936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:56.428617 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvlm4\" (UniqueName: \"kubernetes.io/projected/820af093-87be-4078-9098-1997a629c2c7-kube-api-access-dvlm4\") pod \"payload-pre-processing-e2e-ipp-b-f649b0-75dd767db7-8fdnd\" (UID: \"820af093-87be-4078-9098-1997a629c2c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-75dd767db7-8fdnd" Oct 30 00:27:56.456935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:56.451205 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-75dd767db7-8fdnd" Oct 30 00:27:56.513944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:56.511969 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-75dd767db7-8fdnd"] Oct 30 00:27:56.573086 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:56.573051 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:27:56.573346 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:56.573313 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:27:56.583095 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:56.583051 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:56.621219 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:56.619755 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:56.664848 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:56.664813 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f649b0-57947cf4fd-qdghs"] Oct 30 00:27:56.741207 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:56.735189 5001 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 00:27:56.741207 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:56.735246 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 00:27:56.860941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:56.860138 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 00:27:56.869852 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:56.868978 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 00:27:56.943613 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:56.943063 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:56.968943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:56.949887 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-57947cf4fd-qdghs" Oct 30 00:27:57.005505 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:57.005455 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 00:27:57.005727 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:57.005701 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 00:27:57.103037 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:57.090506 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:27:57.103037 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:57.090559 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f649b0-57947cf4fd-qdghs"] Oct 30 00:27:57.123122 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:57.121988 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:57.123122 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:57.122359 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:27:57.135304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:57.134833 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 00:27:57.135304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:57.134890 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" podUID="4fb15e40-3233-4329-917c-6bfcf33cb894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 00:27:57.142793 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:57.142703 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": context deadline exceeded" Oct 30 00:27:57.197951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:57.178631 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4925fd35-f8b1-4706-990d-a291bb1ed760-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f649b0-57947cf4fd-qdghs\" (UID: \"4925fd35-f8b1-4706-990d-a291bb1ed760\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-57947cf4fd-qdghs" Oct 30 00:27:57.197951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:57.178994 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnjg5\" (UniqueName: \"kubernetes.io/projected/4925fd35-f8b1-4706-990d-a291bb1ed760-kube-api-access-mnjg5\") pod \"payload-processing-e2e-ipp-b-f649b0-57947cf4fd-qdghs\" (UID: \"4925fd35-f8b1-4706-990d-a291bb1ed760\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-57947cf4fd-qdghs" Oct 30 00:27:57.197951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:57.181273 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 00:27:57.197951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:57.181320 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" podUID="34da2ec4-9938-4eaf-9845-dc23bcbc761a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 00:27:57.199576 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:57.198439 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:27:57.275634 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:57.275593 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:57.281391 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:57.281345 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnjg5\" (UniqueName: \"kubernetes.io/projected/4925fd35-f8b1-4706-990d-a291bb1ed760-kube-api-access-mnjg5\") pod \"payload-processing-e2e-ipp-b-f649b0-57947cf4fd-qdghs\" (UID: \"4925fd35-f8b1-4706-990d-a291bb1ed760\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-57947cf4fd-qdghs" Oct 30 00:27:57.281969 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:57.281942 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4925fd35-f8b1-4706-990d-a291bb1ed760-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f649b0-57947cf4fd-qdghs\" (UID: \"4925fd35-f8b1-4706-990d-a291bb1ed760\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-57947cf4fd-qdghs" Oct 30 00:27:57.293958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:57.284543 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4925fd35-f8b1-4706-990d-a291bb1ed760-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f649b0-57947cf4fd-qdghs\" (UID: \"4925fd35-f8b1-4706-990d-a291bb1ed760\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-57947cf4fd-qdghs" Oct 30 00:27:57.361723 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:57.361688 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnjg5\" (UniqueName: \"kubernetes.io/projected/4925fd35-f8b1-4706-990d-a291bb1ed760-kube-api-access-mnjg5\") pod \"payload-processing-e2e-ipp-b-f649b0-57947cf4fd-qdghs\" (UID: \"4925fd35-f8b1-4706-990d-a291bb1ed760\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-57947cf4fd-qdghs" Oct 30 00:27:57.568942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:57.562285 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:27:57.568942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:57.562344 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:27:57.586568 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:57.586000 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:57.627656 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:57.627204 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-57947cf4fd-qdghs" Oct 30 00:27:57.643473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:57.643056 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:57.765080 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:57.762439 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:27:57.765080 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:57.763833 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:27:57.765080 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:57.764723 5001 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 00:27:57.765080 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:57.764770 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 00:27:57.878074 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:57.861575 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:57.878074 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:57.876175 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:57.989000 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:57.945835 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:58.000474 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:57.999790 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 00:27:58.000474 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:57.999847 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 00:27:58.079201 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:58.078656 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 00:27:58.079201 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:58.078711 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" podUID="4fb15e40-3233-4329-917c-6bfcf33cb894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 00:27:58.094149 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:58.080302 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:27:58.123673 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:58.123635 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:58.124747 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:58.124716 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:27:58.175722 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:58.175691 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:27:58.216628 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:58.208029 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:27:58.267177 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:58.263328 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:27:58.267177 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:58.264843 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:27:58.308936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:58.283401 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:58.479136 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:58.479015 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:58.669055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:58.664127 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:58.702009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:58.685235 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:58.807389 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:58.807338 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:27:58.855964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:58.855928 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:27:58.856201 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:58.856170 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:27:58.968936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:58.952727 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:58.982681 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:58.982647 5001 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 00:27:58.984809 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:58.983328 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 00:27:59.164569 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:59.163902 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 00:27:59.164569 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:59.163999 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 00:27:59.332320 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:59.325416 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:59.350939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:59.341992 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:59.679976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:59.669705 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 30 00:27:59.696670 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:59.688078 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:27:59.743965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:59.743926 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:27:59.744230 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:59.744202 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:27:59.758122 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:59.758097 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:27:59.758595 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:59.758576 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:27:59.758666 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:59.758638 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:27:59.758701 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:59.758673 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-75dd767db7-8fdnd"] Oct 30 00:27:59.761141 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:59.761118 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:27:59.774359 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:59.774341 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:27:59.777335 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:59.777315 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:27:59.796333 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:59.795491 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:27:59.802617 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:59.801891 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:27:59.853594 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:27:59.853565 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4925fd35_f8b1_4706_990d_a291bb1ed760.slice/crio-e045aa0493885d3bcdac506ce06ca3b05b507616885911db9fd3a70603f8f40f WatchSource:0}: Error finding container e045aa0493885d3bcdac506ce06ca3b05b507616885911db9fd3a70603f8f40f: Status 404 returned error can't find the container with id e045aa0493885d3bcdac506ce06ca3b05b507616885911db9fd3a70603f8f40f Oct 30 00:27:59.855789 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:27:59.855504 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f649b0-57947cf4fd-qdghs"] Oct 30 00:28:00.001309 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:00.001280 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:28:00.002459 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:00.002434 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:28:00.087299 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:00.087145 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-75dd767db7-8fdnd" event={"ID":"820af093-87be-4078-9098-1997a629c2c7","Type":"ContainerStarted","Data":"1c82807655b4491ce427e14febc1fd7b071eda9eb55763e2d74f3b6fa5cb024e"} Oct 30 00:28:00.087299 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:00.087209 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-75dd767db7-8fdnd" event={"ID":"820af093-87be-4078-9098-1997a629c2c7","Type":"ContainerStarted","Data":"4a678a32acc13c328af0460f8cdbbfee13014a940d1513216cc3450acc3bad96"} Oct 30 00:28:00.087753 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:00.087262 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-75dd767db7-8fdnd" Oct 30 00:28:00.092874 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:00.092835 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-57947cf4fd-qdghs" event={"ID":"4925fd35-f8b1-4706-990d-a291bb1ed760","Type":"ContainerStarted","Data":"86e07d664b557dcfb0583e48d74116053c99e34286cf00a934adf371ef49abcb"} Oct 30 00:28:00.093898 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:00.093874 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-57947cf4fd-qdghs" event={"ID":"4925fd35-f8b1-4706-990d-a291bb1ed760","Type":"ContainerStarted","Data":"e045aa0493885d3bcdac506ce06ca3b05b507616885911db9fd3a70603f8f40f"} Oct 30 00:28:00.094100 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:00.094080 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-57947cf4fd-qdghs" Oct 30 00:28:00.150431 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:00.150369 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-75dd767db7-8fdnd" podStartSLOduration=5.150350004 podStartE2EDuration="5.150350004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:28:00.146528252 +0000 UTC m=+1395.352029434" watchObservedRunningTime="2026-10-30 00:28:00.150350004 +0000 UTC m=+1395.355851186" Oct 30 00:28:00.183143 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:00.183084 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-57947cf4fd-qdghs" podStartSLOduration=5.183067364 podStartE2EDuration="5.183067364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:28:00.18208833 +0000 UTC m=+1395.387589512" watchObservedRunningTime="2026-10-30 00:28:00.183067364 +0000 UTC m=+1395.388568547" Oct 30 00:28:00.584947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:00.561492 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:28:00.625244 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:00.624900 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 00:28:01.564263 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:01.564232 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:28:01.564744 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:01.564730 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:28:01.565556 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:01.565532 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:28:05.636588 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:05.634302 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-5658554c47-sq9k7"] Oct 30 00:28:05.698935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:05.686434 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-5658554c47-sq9k7" Oct 30 00:28:05.737255 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:05.737217 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a15230\"" Oct 30 00:28:05.767731 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:05.767691 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-5658554c47-sq9k7"] Oct 30 00:28:05.785348 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:05.785309 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmrvj\" (UniqueName: \"kubernetes.io/projected/17582778-cc63-4b44-ac34-ee9a4c866bba-kube-api-access-jmrvj\") pod \"payload-pre-processing-e2e-ipp-b-a15230-5658554c47-sq9k7\" (UID: \"17582778-cc63-4b44-ac34-ee9a4c866bba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-5658554c47-sq9k7" Oct 30 00:28:05.797934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:05.787517 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17582778-cc63-4b44-ac34-ee9a4c866bba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a15230-5658554c47-sq9k7\" (UID: \"17582778-cc63-4b44-ac34-ee9a4c866bba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-5658554c47-sq9k7" Oct 30 00:28:05.799641 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:05.799363 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a15230-dockercfg-7qhl7\"" Oct 30 00:28:05.861093 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:05.861040 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:28:05.863348 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:05.863297 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 00:28:05.902548 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:05.902448 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17582778-cc63-4b44-ac34-ee9a4c866bba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a15230-5658554c47-sq9k7\" (UID: \"17582778-cc63-4b44-ac34-ee9a4c866bba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-5658554c47-sq9k7" Oct 30 00:28:05.902844 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:05.902821 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmrvj\" (UniqueName: \"kubernetes.io/projected/17582778-cc63-4b44-ac34-ee9a4c866bba-kube-api-access-jmrvj\") pod \"payload-pre-processing-e2e-ipp-b-a15230-5658554c47-sq9k7\" (UID: \"17582778-cc63-4b44-ac34-ee9a4c866bba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-5658554c47-sq9k7" Oct 30 00:28:05.904682 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:05.904586 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17582778-cc63-4b44-ac34-ee9a4c866bba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a15230-5658554c47-sq9k7\" (UID: \"17582778-cc63-4b44-ac34-ee9a4c866bba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-5658554c47-sq9k7" Oct 30 00:28:05.956222 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:05.956140 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmrvj\" (UniqueName: \"kubernetes.io/projected/17582778-cc63-4b44-ac34-ee9a4c866bba-kube-api-access-jmrvj\") pod \"payload-pre-processing-e2e-ipp-b-a15230-5658554c47-sq9k7\" (UID: \"17582778-cc63-4b44-ac34-ee9a4c866bba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-5658554c47-sq9k7" Oct 30 00:28:06.067343 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:06.066014 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:28:06.123853 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:06.123016 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:28:06.123853 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:06.123602 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-5658554c47-sq9k7" Oct 30 00:28:06.188832 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:06.186004 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:28:06.188832 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:06.187593 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:28:06.594245 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:06.594192 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:28:06.646239 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:06.642113 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:28:06.870996 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:06.864723 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 00:28:06.870996 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:06.866362 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:28:06.887933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:06.874217 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:28:06.887933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:06.874770 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:28:07.128581 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:07.126056 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:28:07.128581 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:07.126512 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:28:07.128581 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:07.127714 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:28:07.139632 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:07.139596 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:28:07.148016 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:07.140011 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": context deadline exceeded" Oct 30 00:28:07.227945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:07.210198 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:28:07.227945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:07.210758 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:28:07.613940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:07.607890 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:28:07.639958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:07.625286 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:28:07.642373 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:07.642326 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:28:07.685176 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:07.685129 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:28:07.875346 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:07.875256 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:28:07.926007 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:07.877130 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 00:28:08.088938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:08.035011 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:28:08.152730 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:08.152647 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:28:08.184940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:08.157322 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:28:08.275944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:08.274892 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:28:08.297934 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:28:08.290869 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17582778_cc63_4b44_ac34_ee9a4c866bba.slice/crio-624fc06b78a59d156542cfd992d14d600793f458e8eaef82345bc54a2ec697da WatchSource:0}: Error finding container 624fc06b78a59d156542cfd992d14d600793f458e8eaef82345bc54a2ec697da: Status 404 returned error can't find the container with id 624fc06b78a59d156542cfd992d14d600793f458e8eaef82345bc54a2ec697da Oct 30 00:28:08.655999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:08.655259 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:28:08.828934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:08.821217 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:28:08.847241 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:08.833353 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-5658554c47-sq9k7" event={"ID":"17582778-cc63-4b44-ac34-ee9a4c866bba","Type":"ContainerStarted","Data":"624fc06b78a59d156542cfd992d14d600793f458e8eaef82345bc54a2ec697da"} Oct 30 00:28:08.913413 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:08.913330 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:28:08.926447 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:08.926409 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:28:08.966080 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:08.966017 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 00:28:09.347290 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:09.347238 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:28:09.364026 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:09.363991 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-5658554c47-sq9k7"] Oct 30 00:28:09.462042 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:09.461578 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:28:09.669074 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:09.668042 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:28:09.795950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:09.794475 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:28:09.795950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:09.795025 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:28:09.797129 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:09.797102 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:28:09.799557 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:09.799536 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:28:10.152368 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:10.140627 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-5658554c47-sq9k7" event={"ID":"17582778-cc63-4b44-ac34-ee9a4c866bba","Type":"ContainerStarted","Data":"827f8f579ad97a31a23a6e223c696bf65f534a64bf1811da87bec5389984bd59"} Oct 30 00:28:10.165945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:10.152940 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-5658554c47-sq9k7" Oct 30 00:28:10.808252 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:10.808190 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-5658554c47-sq9k7" podStartSLOduration=6.808171784 podStartE2EDuration="6.808171784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:28:10.218635337 +0000 UTC m=+1405.424136536" watchObservedRunningTime="2026-10-30 00:28:10.808171784 +0000 UTC m=+1406.013672966" Oct 30 00:28:10.983948 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:10.973008 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:28:11.059358 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:11.059261 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:28:11.062823 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:11.062796 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35b82218-88cpk"] Oct 30 00:28:11.099508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:11.099450 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35b82218-88cpk"] Oct 30 00:28:11.105129 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:11.105092 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-75dd767db7-8fdnd" Oct 30 00:28:12.122933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:12.122155 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-57947cf4fd-qdghs" Oct 30 00:28:12.990617 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:12.990584 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f086c625-6fda-4000-9abf-1055f720d140" path="/var/lib/kubelet/pods/f086c625-6fda-4000-9abf-1055f720d140/volumes" Oct 30 00:28:15.150114 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:15.150062 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cfcf7-bm58n"] Oct 30 00:28:15.159980 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:15.159904 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cfcf7-bm58n" Oct 30 00:28:15.182174 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:15.180311 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:28:15.182174 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:15.180943 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:28:15.182174 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:15.181026 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:28:15.194558 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:15.194506 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:28:15.199547 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:15.197983 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cfcf7-bm58n"] Oct 30 00:28:15.342537 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:15.342490 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a243f64a-3cd5-46b4-8f02-27b25c1d6e87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6cfcf7-bm58n\" (UID: \"a243f64a-3cd5-46b4-8f02-27b25c1d6e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cfcf7-bm58n" Oct 30 00:28:15.342713 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:15.342572 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a243f64a-3cd5-46b4-8f02-27b25c1d6e87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6cfcf7-bm58n\" (UID: \"a243f64a-3cd5-46b4-8f02-27b25c1d6e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cfcf7-bm58n" Oct 30 00:28:15.342713 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:15.342679 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55k8d\" (UniqueName: \"kubernetes.io/projected/a243f64a-3cd5-46b4-8f02-27b25c1d6e87-kube-api-access-55k8d\") pod \"maas-api-revoke-keys-e2e-api-a-6cfcf7-bm58n\" (UID: \"a243f64a-3cd5-46b4-8f02-27b25c1d6e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cfcf7-bm58n" Oct 30 00:28:15.445748 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:15.444386 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55k8d\" (UniqueName: \"kubernetes.io/projected/a243f64a-3cd5-46b4-8f02-27b25c1d6e87-kube-api-access-55k8d\") pod \"maas-api-revoke-keys-e2e-api-a-6cfcf7-bm58n\" (UID: \"a243f64a-3cd5-46b4-8f02-27b25c1d6e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cfcf7-bm58n" Oct 30 00:28:15.445748 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:15.444692 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a243f64a-3cd5-46b4-8f02-27b25c1d6e87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6cfcf7-bm58n\" (UID: \"a243f64a-3cd5-46b4-8f02-27b25c1d6e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cfcf7-bm58n" Oct 30 00:28:15.445748 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:15.444755 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a243f64a-3cd5-46b4-8f02-27b25c1d6e87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6cfcf7-bm58n\" (UID: \"a243f64a-3cd5-46b4-8f02-27b25c1d6e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cfcf7-bm58n" Oct 30 00:28:15.445748 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:15.445648 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a243f64a-3cd5-46b4-8f02-27b25c1d6e87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6cfcf7-bm58n\" (UID: \"a243f64a-3cd5-46b4-8f02-27b25c1d6e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cfcf7-bm58n" Oct 30 00:28:15.448022 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:15.447986 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a243f64a-3cd5-46b4-8f02-27b25c1d6e87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6cfcf7-bm58n\" (UID: \"a243f64a-3cd5-46b4-8f02-27b25c1d6e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cfcf7-bm58n" Oct 30 00:28:15.466295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:15.466261 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55k8d\" (UniqueName: \"kubernetes.io/projected/a243f64a-3cd5-46b4-8f02-27b25c1d6e87-kube-api-access-55k8d\") pod \"maas-api-revoke-keys-e2e-api-a-6cfcf7-bm58n\" (UID: \"a243f64a-3cd5-46b4-8f02-27b25c1d6e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cfcf7-bm58n" Oct 30 00:28:15.502545 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:15.502261 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cfcf7-bm58n" Oct 30 00:28:15.832453 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:28:15.832414 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda243f64a_3cd5_46b4_8f02_27b25c1d6e87.slice/crio-f3971f55fd03e99a9cd3a57e78595f743e707ca635bddfd972cf85652e8ca2b7 WatchSource:0}: Error finding container f3971f55fd03e99a9cd3a57e78595f743e707ca635bddfd972cf85652e8ca2b7: Status 404 returned error can't find the container with id f3971f55fd03e99a9cd3a57e78595f743e707ca635bddfd972cf85652e8ca2b7 Oct 30 00:28:15.832632 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:15.832611 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cfcf7-bm58n"] Oct 30 00:28:16.407078 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:16.399341 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cfcf7-bm58n" event={"ID":"a243f64a-3cd5-46b4-8f02-27b25c1d6e87","Type":"ContainerStarted","Data":"f3971f55fd03e99a9cd3a57e78595f743e707ca635bddfd972cf85652e8ca2b7"} Oct 30 00:28:17.328011 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:28:17.327928 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:28:17.410846 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:17.410808 5001 generic.go:356] "Generic (PLEG): container finished" podID="a243f64a-3cd5-46b4-8f02-27b25c1d6e87" containerID="c3f694f3e3083c848b84982973eb54a7c2f6b48458a0dbbca9b0302250e74c69" exitCode=0 Oct 30 00:28:17.411268 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:17.410866 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cfcf7-bm58n" event={"ID":"a243f64a-3cd5-46b4-8f02-27b25c1d6e87","Type":"ContainerDied","Data":"c3f694f3e3083c848b84982973eb54a7c2f6b48458a0dbbca9b0302250e74c69"} Oct 30 00:28:18.464101 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:18.464060 5001 patch_prober.go:29] interesting pod/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:28:18.464466 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:18.464129 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" podUID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:28:18.587136 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:18.587110 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f649b0-57947cf4fd-qdghs_4925fd35-f8b1-4706-990d-a291bb1ed760/payload-processing/0.log" Oct 30 00:28:18.621217 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:18.621185 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cfcf7-bm58n" Oct 30 00:28:18.721502 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:18.721428 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a243f64a-3cd5-46b4-8f02-27b25c1d6e87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a243f64a-3cd5-46b4-8f02-27b25c1d6e87-maas-api-openshift-ca\") pod \"a243f64a-3cd5-46b4-8f02-27b25c1d6e87\" (UID: \"a243f64a-3cd5-46b4-8f02-27b25c1d6e87\") " Oct 30 00:28:18.721695 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:18.721500 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a243f64a-3cd5-46b4-8f02-27b25c1d6e87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a243f64a-3cd5-46b4-8f02-27b25c1d6e87-maas-api-xks-ca\") pod \"a243f64a-3cd5-46b4-8f02-27b25c1d6e87\" (UID: \"a243f64a-3cd5-46b4-8f02-27b25c1d6e87\") " Oct 30 00:28:18.721695 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:18.721578 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a243f64a-3cd5-46b4-8f02-27b25c1d6e87-kube-api-access-55k8d\" (UniqueName: \"kubernetes.io/projected/a243f64a-3cd5-46b4-8f02-27b25c1d6e87-kube-api-access-55k8d\") pod \"a243f64a-3cd5-46b4-8f02-27b25c1d6e87\" (UID: \"a243f64a-3cd5-46b4-8f02-27b25c1d6e87\") " Oct 30 00:28:18.722961 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:18.722904 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a243f64a-3cd5-46b4-8f02-27b25c1d6e87-maas-api-openshift-ca" pod "a243f64a-3cd5-46b4-8f02-27b25c1d6e87" (UID: "a243f64a-3cd5-46b4-8f02-27b25c1d6e87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:28:18.724683 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:18.724630 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a243f64a-3cd5-46b4-8f02-27b25c1d6e87-maas-api-xks-ca" pod "a243f64a-3cd5-46b4-8f02-27b25c1d6e87" (UID: "a243f64a-3cd5-46b4-8f02-27b25c1d6e87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:28:18.724683 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:18.724672 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a243f64a-3cd5-46b4-8f02-27b25c1d6e87-kube-api-access-55k8d" pod "a243f64a-3cd5-46b4-8f02-27b25c1d6e87" (UID: "a243f64a-3cd5-46b4-8f02-27b25c1d6e87"). InnerVolumeSpecName "kube-api-access-55k8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:28:18.824453 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:18.824424 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a243f64a-3cd5-46b4-8f02-27b25c1d6e87-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:18.824453 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:18.824454 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a243f64a-3cd5-46b4-8f02-27b25c1d6e87-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:20.050429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:18.824464 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55k8d\" (UniqueName: \"kubernetes.io/projected/a243f64a-3cd5-46b4-8f02-27b25c1d6e87-kube-api-access-55k8d\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:20.050429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:19.474207 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cfcf7-bm58n" event={"ID":"a243f64a-3cd5-46b4-8f02-27b25c1d6e87","Type":"ContainerDied","Data":"f3971f55fd03e99a9cd3a57e78595f743e707ca635bddfd972cf85652e8ca2b7"} Oct 30 00:28:20.050429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:19.474302 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3971f55fd03e99a9cd3a57e78595f743e707ca635bddfd972cf85652e8ca2b7" Oct 30 00:28:20.050429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:19.474368 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cfcf7-bm58n" Oct 30 00:28:20.373335 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.373239 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v"] Oct 30 00:28:20.374513 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.374474 5001 cpu_manager.go:414] "containerMap: removing container" podUID="a243f64a-3cd5-46b4-8f02-27b25c1d6e87" containerName="revoke-keys" Oct 30 00:28:20.374513 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.374509 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a243f64a-3cd5-46b4-8f02-27b25c1d6e87" containerName="revoke-keys" Oct 30 00:28:20.374787 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.374732 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a243f64a-3cd5-46b4-8f02-27b25c1d6e87" containerName="revoke-keys" Oct 30 00:28:20.378614 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.378589 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" Oct 30 00:28:20.399418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.399390 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v"] Oct 30 00:28:20.401010 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.400988 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-3fb3ad\"/\"ipp-route-3fb3ad-kserve-self-signed-certs\"" Oct 30 00:28:20.401165 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.401077 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-3fb3ad\"/\"default-dockercfg-pvgxb\"" Oct 30 00:28:20.402569 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.401300 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-3fb3ad\"/\"openshift-service-ca.crt\"" Oct 30 00:28:20.402569 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.401373 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-3fb3ad\"/\"kube-root-ca.crt\"" Oct 30 00:28:20.485344 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.485310 5001 generic.go:356] "Generic (PLEG): container finished" podID="a684c9be-f1fe-420b-a3a0-16ec60772131" containerID="06ebf0b626bf6de28c6b1d380ef384c11f214f1de1bab6ea1cac5ce013619781" exitCode=0 Oct 30 00:28:20.485560 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.485379 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" event={"ID":"a684c9be-f1fe-420b-a3a0-16ec60772131","Type":"ContainerDied","Data":"06ebf0b626bf6de28c6b1d380ef384c11f214f1de1bab6ea1cac5ce013619781"} Oct 30 00:28:20.485948 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.485896 5001 scope.go:122] "RemoveContainer" containerID="06ebf0b626bf6de28c6b1d380ef384c11f214f1de1bab6ea1cac5ce013619781" Oct 30 00:28:20.487612 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.487591 5001 generic.go:356] "Generic (PLEG): container finished" podID="91eb024d-d850-4682-9821-bcae4cb40b49" containerID="8eacc60f7c5c21d0a438dc1a6137c99db7dddde626d7aa5a33f5807edbde3bfa" exitCode=0 Oct 30 00:28:20.487722 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.487662 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" event={"ID":"91eb024d-d850-4682-9821-bcae4cb40b49","Type":"ContainerDied","Data":"8eacc60f7c5c21d0a438dc1a6137c99db7dddde626d7aa5a33f5807edbde3bfa"} Oct 30 00:28:20.488162 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.488142 5001 scope.go:122] "RemoveContainer" containerID="8eacc60f7c5c21d0a438dc1a6137c99db7dddde626d7aa5a33f5807edbde3bfa" Oct 30 00:28:20.488283 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.488242 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:28:20.488356 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.488324 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:28:20.488418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.488359 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:28:20.489950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.489894 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:28:20.490041 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.490006 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:28:20.490085 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.490044 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:28:20.490906 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.490873 5001 generic.go:356] "Generic (PLEG): container finished" podID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerID="1604d7cfa69887eecbbdf6d7f18b6f4d47c680b9ef80fce3e6478b12a59d7423" exitCode=0 Oct 30 00:28:20.491042 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.490956 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" event={"ID":"dfe41eb3-2947-447a-aa04-4eca33589f1d","Type":"ContainerDied","Data":"1604d7cfa69887eecbbdf6d7f18b6f4d47c680b9ef80fce3e6478b12a59d7423"} Oct 30 00:28:20.491394 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.491375 5001 scope.go:122] "RemoveContainer" containerID="1604d7cfa69887eecbbdf6d7f18b6f4d47c680b9ef80fce3e6478b12a59d7423" Oct 30 00:28:20.501354 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.501319 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:28:20.501452 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.501403 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:28:20.501511 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.501447 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:28:20.519697 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.519663 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-model-cache\") pod \"ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v\" (UID: \"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e\") " pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" Oct 30 00:28:20.519799 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.519716 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-dshm\") pod \"ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v\" (UID: \"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e\") " pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" Oct 30 00:28:20.519873 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.519837 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdskl\" (UniqueName: \"kubernetes.io/projected/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-kube-api-access-cdskl\") pod \"ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v\" (UID: \"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e\") " pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" Oct 30 00:28:20.519949 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.519888 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-home\") pod \"ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v\" (UID: \"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e\") " pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" Oct 30 00:28:20.520084 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.520061 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-tls-certs\") pod \"ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v\" (UID: \"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e\") " pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" Oct 30 00:28:20.520177 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.520114 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-tmp-dir\") pod \"ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v\" (UID: \"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e\") " pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" Oct 30 00:28:20.652239 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.649675 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-model-cache\") pod \"ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v\" (UID: \"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e\") " pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" Oct 30 00:28:20.652239 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.649757 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-dshm\") pod \"ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v\" (UID: \"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e\") " pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" Oct 30 00:28:20.652239 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.649844 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdskl\" (UniqueName: \"kubernetes.io/projected/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-kube-api-access-cdskl\") pod \"ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v\" (UID: \"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e\") " pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" Oct 30 00:28:20.652239 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.649899 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-home\") pod \"ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v\" (UID: \"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e\") " pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" Oct 30 00:28:20.652239 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.650266 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-tls-certs\") pod \"ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v\" (UID: \"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e\") " pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" Oct 30 00:28:20.652239 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.650328 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-tmp-dir\") pod \"ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v\" (UID: \"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e\") " pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" Oct 30 00:28:20.652239 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.652069 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-model-cache\") pod \"ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v\" (UID: \"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e\") " pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" Oct 30 00:28:20.660948 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.656003 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-home\") pod \"ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v\" (UID: \"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e\") " pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" Oct 30 00:28:20.708033 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.704952 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-dshm\") pod \"ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v\" (UID: \"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e\") " pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" Oct 30 00:28:20.708033 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.705584 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-tls-certs\") pod \"ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v\" (UID: \"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e\") " pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" Oct 30 00:28:20.708033 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.705701 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-tmp-dir\") pod \"ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v\" (UID: \"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e\") " pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" Oct 30 00:28:20.747187 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:20.744612 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdskl\" (UniqueName: \"kubernetes.io/projected/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-kube-api-access-cdskl\") pod \"ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v\" (UID: \"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e\") " pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" Oct 30 00:28:21.008728 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:21.008639 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" Oct 30 00:28:21.308958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:21.300600 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-5658554c47-sq9k7" Oct 30 00:28:21.502251 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:21.502206 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" event={"ID":"dfe41eb3-2947-447a-aa04-4eca33589f1d","Type":"ContainerStarted","Data":"9a852f3a59bf02b2bf7e1dd63e3aedacc9d8dc7e0e6ab0886b09209460f4f5e5"} Oct 30 00:28:21.505434 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:21.505405 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" event={"ID":"a684c9be-f1fe-420b-a3a0-16ec60772131","Type":"ContainerStarted","Data":"99b9fd83eeb52926a660a6e40e04bacc5ff3cabe8fcfc9b73ecae525375b802e"} Oct 30 00:28:21.508619 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:21.508591 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" event={"ID":"91eb024d-d850-4682-9821-bcae4cb40b49","Type":"ContainerStarted","Data":"197e965431543abb5219476fdc6bc90d090107bf85c869b387b45d177bde1651"} Oct 30 00:28:21.528600 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:21.527963 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v"] Oct 30 00:28:21.537388 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:28:21.537279 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27fd701d_57b6_4ea6_a0ff_b8cb8acb5c5e.slice/crio-e10024d050750583165eb51e835251e7d8b88e013b1a664be6267f9022cd5094 WatchSource:0}: Error finding container e10024d050750583165eb51e835251e7d8b88e013b1a664be6267f9022cd5094: Status 404 returned error can't find the container with id e10024d050750583165eb51e835251e7d8b88e013b1a664be6267f9022cd5094 Oct 30 00:28:21.676096 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:21.676059 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:28:21.677347 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:21.677278 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 00:28:21.677505 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:21.677375 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" podUID="91eb024d-d850-4682-9821-bcae4cb40b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 00:28:22.271938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:22.271237 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:28:22.271938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:22.271379 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 00:28:22.271938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:22.271422 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" podUID="a684c9be-f1fe-420b-a3a0-16ec60772131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 00:28:22.291932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:22.287964 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8b3bf0-729qb"] Oct 30 00:28:22.317934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:22.312197 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8b3bf0-729qb" Oct 30 00:28:22.337935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:22.328867 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:28:22.337935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:22.329024 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:28:22.370936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:22.369035 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:28:22.382935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:22.382699 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:28:22.382935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:22.382808 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:28:22.441939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:22.417325 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:28:22.441939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:22.417460 5001 patch_prober.go:29] interesting pod/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 00:28:22.441939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:22.417505 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" podUID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 00:28:22.441939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:22.429516 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8b3bf0-729qb"] Oct 30 00:28:22.572036 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:22.538017 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m25vv\" (UniqueName: \"kubernetes.io/projected/28c5ba02-cec3-4d4d-b0bf-67c2db6b981d-kube-api-access-m25vv\") pod \"maas-api-revoke-keys-e2e-praxis-a-8b3bf0-729qb\" (UID: \"28c5ba02-cec3-4d4d-b0bf-67c2db6b981d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8b3bf0-729qb" Oct 30 00:28:22.572036 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:22.538140 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28c5ba02-cec3-4d4d-b0bf-67c2db6b981d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-8b3bf0-729qb\" (UID: \"28c5ba02-cec3-4d4d-b0bf-67c2db6b981d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8b3bf0-729qb" Oct 30 00:28:22.572036 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:22.538199 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28c5ba02-cec3-4d4d-b0bf-67c2db6b981d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-8b3bf0-729qb\" (UID: \"28c5ba02-cec3-4d4d-b0bf-67c2db6b981d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8b3bf0-729qb" Oct 30 00:28:22.572036 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:22.567242 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" event={"ID":"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e","Type":"ContainerStarted","Data":"cfa161992b61f7c996feb50568f39a57791397e8941cf89f5f1dbbdcec126604"} Oct 30 00:28:22.572036 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:22.567285 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" event={"ID":"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e","Type":"ContainerStarted","Data":"e10024d050750583165eb51e835251e7d8b88e013b1a664be6267f9022cd5094"} Oct 30 00:28:22.708062 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:22.696133 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28c5ba02-cec3-4d4d-b0bf-67c2db6b981d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-8b3bf0-729qb\" (UID: \"28c5ba02-cec3-4d4d-b0bf-67c2db6b981d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8b3bf0-729qb" Oct 30 00:28:22.708062 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:22.696597 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m25vv\" (UniqueName: \"kubernetes.io/projected/28c5ba02-cec3-4d4d-b0bf-67c2db6b981d-kube-api-access-m25vv\") pod \"maas-api-revoke-keys-e2e-praxis-a-8b3bf0-729qb\" (UID: \"28c5ba02-cec3-4d4d-b0bf-67c2db6b981d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8b3bf0-729qb" Oct 30 00:28:22.708062 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:22.696804 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28c5ba02-cec3-4d4d-b0bf-67c2db6b981d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-8b3bf0-729qb\" (UID: \"28c5ba02-cec3-4d4d-b0bf-67c2db6b981d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8b3bf0-729qb" Oct 30 00:28:22.709264 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:22.709234 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28c5ba02-cec3-4d4d-b0bf-67c2db6b981d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-8b3bf0-729qb\" (UID: \"28c5ba02-cec3-4d4d-b0bf-67c2db6b981d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8b3bf0-729qb" Oct 30 00:28:22.712790 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:22.709945 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 00:28:22.713056 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:22.712991 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" podUID="91eb024d-d850-4682-9821-bcae4cb40b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 00:28:22.741701 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:22.741291 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28c5ba02-cec3-4d4d-b0bf-67c2db6b981d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-8b3bf0-729qb\" (UID: \"28c5ba02-cec3-4d4d-b0bf-67c2db6b981d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8b3bf0-729qb" Oct 30 00:28:22.778972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:22.769832 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" podStartSLOduration=2.76980864 podStartE2EDuration="2.76980864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:28:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:28:22.693813742 +0000 UTC m=+1417.899314927" watchObservedRunningTime="2026-10-30 00:28:22.76980864 +0000 UTC m=+1417.975309823" Oct 30 00:28:22.778972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:22.776729 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m25vv\" (UniqueName: \"kubernetes.io/projected/28c5ba02-cec3-4d4d-b0bf-67c2db6b981d-kube-api-access-m25vv\") pod \"maas-api-revoke-keys-e2e-praxis-a-8b3bf0-729qb\" (UID: \"28c5ba02-cec3-4d4d-b0bf-67c2db6b981d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8b3bf0-729qb" Oct 30 00:28:23.009530 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:23.009405 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" Oct 30 00:28:23.032477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:23.032187 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8b3bf0-729qb" Oct 30 00:28:23.289937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:23.287688 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 00:28:23.289937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:23.287749 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" podUID="a684c9be-f1fe-420b-a3a0-16ec60772131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 00:28:23.541176 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:23.502412 5001 patch_prober.go:29] interesting pod/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 00:28:23.541176 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:23.502472 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" podUID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 00:28:23.677937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:23.677637 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 00:28:23.677937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:23.677695 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" podUID="91eb024d-d850-4682-9821-bcae4cb40b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 00:28:23.901088 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:23.901061 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8b3bf0-729qb"] Oct 30 00:28:24.194081 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:24.193981 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f8397b-s8hxd"] Oct 30 00:28:24.297289 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:24.297256 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 00:28:24.297473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:24.297445 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" podUID="a684c9be-f1fe-420b-a3a0-16ec60772131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 00:28:24.395255 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:24.395212 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:28:24.395415 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:24.395267 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f8397b-s8hxd"] Oct 30 00:28:24.395591 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:24.395571 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f8397b-s8hxd" Oct 30 00:28:24.426263 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:24.426228 5001 patch_prober.go:29] interesting pod/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 00:28:24.426546 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:24.426513 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" podUID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 00:28:24.505827 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:24.505794 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x76bc\" (UniqueName: \"kubernetes.io/projected/c0a4294e-62aa-49ea-928a-da94c5381802-kube-api-access-x76bc\") pod \"maas-api-revoke-keys-e2e-ipp-a-f8397b-s8hxd\" (UID: \"c0a4294e-62aa-49ea-928a-da94c5381802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f8397b-s8hxd" Oct 30 00:28:24.506005 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:24.505835 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a4294e-62aa-49ea-928a-da94c5381802-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f8397b-s8hxd\" (UID: \"c0a4294e-62aa-49ea-928a-da94c5381802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f8397b-s8hxd" Oct 30 00:28:24.506005 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:24.505903 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a4294e-62aa-49ea-928a-da94c5381802-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f8397b-s8hxd\" (UID: \"c0a4294e-62aa-49ea-928a-da94c5381802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f8397b-s8hxd" Oct 30 00:28:24.607632 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:24.607583 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a4294e-62aa-49ea-928a-da94c5381802-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f8397b-s8hxd\" (UID: \"c0a4294e-62aa-49ea-928a-da94c5381802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f8397b-s8hxd" Oct 30 00:28:24.608172 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:24.607983 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x76bc\" (UniqueName: \"kubernetes.io/projected/c0a4294e-62aa-49ea-928a-da94c5381802-kube-api-access-x76bc\") pod \"maas-api-revoke-keys-e2e-ipp-a-f8397b-s8hxd\" (UID: \"c0a4294e-62aa-49ea-928a-da94c5381802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f8397b-s8hxd" Oct 30 00:28:24.608172 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:24.608030 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a4294e-62aa-49ea-928a-da94c5381802-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f8397b-s8hxd\" (UID: \"c0a4294e-62aa-49ea-928a-da94c5381802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f8397b-s8hxd" Oct 30 00:28:24.628935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:24.608858 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a4294e-62aa-49ea-928a-da94c5381802-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f8397b-s8hxd\" (UID: \"c0a4294e-62aa-49ea-928a-da94c5381802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f8397b-s8hxd" Oct 30 00:28:24.680950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:24.670099 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x76bc\" (UniqueName: \"kubernetes.io/projected/c0a4294e-62aa-49ea-928a-da94c5381802-kube-api-access-x76bc\") pod \"maas-api-revoke-keys-e2e-ipp-a-f8397b-s8hxd\" (UID: \"c0a4294e-62aa-49ea-928a-da94c5381802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f8397b-s8hxd" Oct 30 00:28:24.695671 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:24.694561 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a4294e-62aa-49ea-928a-da94c5381802-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f8397b-s8hxd\" (UID: \"c0a4294e-62aa-49ea-928a-da94c5381802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f8397b-s8hxd" Oct 30 00:28:24.695671 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:24.694703 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 00:28:24.695671 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:24.694763 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" podUID="91eb024d-d850-4682-9821-bcae4cb40b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 00:28:24.700933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:24.700787 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8b3bf0-729qb" event={"ID":"28c5ba02-cec3-4d4d-b0bf-67c2db6b981d","Type":"ContainerStarted","Data":"831922d7fc804959e3c0e4842d19d6807c0ba0e0d0f48ccc6d4b7553e8b4e727"} Oct 30 00:28:24.795823 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:24.795136 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f8397b-s8hxd" Oct 30 00:28:25.206687 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:25.206647 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bb16a8-hgdpt"] Oct 30 00:28:25.234936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:25.230145 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bb16a8-hgdpt" Oct 30 00:28:25.284786 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:25.284748 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 00:28:25.285085 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:25.285052 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" podUID="a684c9be-f1fe-420b-a3a0-16ec60772131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 00:28:25.303194 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:25.303150 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bb16a8-hgdpt"] Oct 30 00:28:25.393987 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:25.371437 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cvwd\" (UniqueName: \"kubernetes.io/projected/d0d7bfe5-4113-4a81-8015-936389553f69-kube-api-access-9cvwd\") pod \"maas-api-revoke-keys-e2e-praxis-a-bb16a8-hgdpt\" (UID: \"d0d7bfe5-4113-4a81-8015-936389553f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bb16a8-hgdpt" Oct 30 00:28:25.393987 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:25.371581 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0d7bfe5-4113-4a81-8015-936389553f69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-bb16a8-hgdpt\" (UID: \"d0d7bfe5-4113-4a81-8015-936389553f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bb16a8-hgdpt" Oct 30 00:28:25.393987 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:25.371766 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0d7bfe5-4113-4a81-8015-936389553f69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-bb16a8-hgdpt\" (UID: \"d0d7bfe5-4113-4a81-8015-936389553f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bb16a8-hgdpt" Oct 30 00:28:25.471923 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:25.471408 5001 patch_prober.go:29] interesting pod/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 00:28:25.471923 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:25.471467 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" podUID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 00:28:25.501950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:25.491811 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0d7bfe5-4113-4a81-8015-936389553f69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-bb16a8-hgdpt\" (UID: \"d0d7bfe5-4113-4a81-8015-936389553f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bb16a8-hgdpt" Oct 30 00:28:25.501950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:25.492093 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cvwd\" (UniqueName: \"kubernetes.io/projected/d0d7bfe5-4113-4a81-8015-936389553f69-kube-api-access-9cvwd\") pod \"maas-api-revoke-keys-e2e-praxis-a-bb16a8-hgdpt\" (UID: \"d0d7bfe5-4113-4a81-8015-936389553f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bb16a8-hgdpt" Oct 30 00:28:25.501950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:25.492205 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0d7bfe5-4113-4a81-8015-936389553f69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-bb16a8-hgdpt\" (UID: \"d0d7bfe5-4113-4a81-8015-936389553f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bb16a8-hgdpt" Oct 30 00:28:25.501950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:25.493311 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0d7bfe5-4113-4a81-8015-936389553f69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-bb16a8-hgdpt\" (UID: \"d0d7bfe5-4113-4a81-8015-936389553f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bb16a8-hgdpt" Oct 30 00:28:25.501950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:25.496323 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0d7bfe5-4113-4a81-8015-936389553f69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-bb16a8-hgdpt\" (UID: \"d0d7bfe5-4113-4a81-8015-936389553f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bb16a8-hgdpt" Oct 30 00:28:25.606268 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:25.606238 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cvwd\" (UniqueName: \"kubernetes.io/projected/d0d7bfe5-4113-4a81-8015-936389553f69-kube-api-access-9cvwd\") pod \"maas-api-revoke-keys-e2e-praxis-a-bb16a8-hgdpt\" (UID: \"d0d7bfe5-4113-4a81-8015-936389553f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bb16a8-hgdpt" Oct 30 00:28:25.689537 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:25.689409 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 00:28:25.691585 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:25.690067 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" podUID="91eb024d-d850-4682-9821-bcae4cb40b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 00:28:26.046935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:26.029751 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bb16a8-hgdpt" Oct 30 00:28:26.310987 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:26.308138 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 00:28:26.310987 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:26.308201 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" podUID="a684c9be-f1fe-420b-a3a0-16ec60772131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 00:28:26.448102 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:26.448048 5001 patch_prober.go:29] interesting pod/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 00:28:26.448195 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:26.448127 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" podUID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 00:28:26.561643 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:26.550096 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f8397b-s8hxd"] Oct 30 00:28:26.706146 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:26.695070 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:28:26.731948 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:26.730392 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8b3bf0-729qb" event={"ID":"28c5ba02-cec3-4d4d-b0bf-67c2db6b981d","Type":"ContainerStarted","Data":"8bd58dae50647ae9a0e5cf08f38a2c23dea86049ad08365a933028d61635c2b7"} Oct 30 00:28:26.735218 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:26.734659 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 00:28:26.735218 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:26.734710 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" podUID="91eb024d-d850-4682-9821-bcae4cb40b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 00:28:26.988351 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:26.968163 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:28:27.089024 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:27.088031 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:28:27.201101 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:27.201054 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:28:27.213028 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:27.205006 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:28:27.269400 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:27.240983 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:28:27.302938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:27.278288 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 00:28:27.358936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:27.339201 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" podUID="a684c9be-f1fe-420b-a3a0-16ec60772131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 00:28:27.501089 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:27.501057 5001 patch_prober.go:29] interesting pod/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 00:28:27.501307 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:27.501278 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" podUID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 00:28:27.595934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:27.576718 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bb16a8-hgdpt"] Oct 30 00:28:27.700389 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:27.689681 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 00:28:27.700389 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:27.689743 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" podUID="91eb024d-d850-4682-9821-bcae4cb40b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 00:28:28.127043 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:28.107758 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bb16a8-hgdpt" event={"ID":"d0d7bfe5-4113-4a81-8015-936389553f69","Type":"ContainerStarted","Data":"41c7763a04c87fd46b7bdd27514c17d28acf57f0fff96e5a2c0d38b66c5cf8aa"} Oct 30 00:28:28.168578 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:28.168536 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f8397b-s8hxd" event={"ID":"c0a4294e-62aa-49ea-928a-da94c5381802","Type":"ContainerStarted","Data":"4b415efaf12f942f8bab78b9afc96e6710f6e1a1c22433f5095a3738b34118ab"} Oct 30 00:28:28.251554 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:28.251515 5001 generic.go:356] "Generic (PLEG): container finished" podID="28c5ba02-cec3-4d4d-b0bf-67c2db6b981d" containerID="8bd58dae50647ae9a0e5cf08f38a2c23dea86049ad08365a933028d61635c2b7" exitCode=0 Oct 30 00:28:28.252141 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:28.252109 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8b3bf0-729qb" event={"ID":"28c5ba02-cec3-4d4d-b0bf-67c2db6b981d","Type":"ContainerDied","Data":"8bd58dae50647ae9a0e5cf08f38a2c23dea86049ad08365a933028d61635c2b7"} Oct 30 00:28:28.269462 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:28.269422 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 00:28:28.269777 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:28.269727 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" podUID="a684c9be-f1fe-420b-a3a0-16ec60772131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 00:28:28.280663 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:28.280597 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f8397b-s8hxd" podStartSLOduration=4.280575286 podStartE2EDuration="4.280575286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:28:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:28:28.214184047 +0000 UTC m=+1423.419685277" watchObservedRunningTime="2026-10-30 00:28:28.280575286 +0000 UTC m=+1423.486076471" Oct 30 00:28:28.422542 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:28.422444 5001 patch_prober.go:29] interesting pod/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 00:28:28.422781 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:28.422750 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" podUID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 00:28:28.693501 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:28.693406 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 00:28:28.693501 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:28.693466 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" podUID="91eb024d-d850-4682-9821-bcae4cb40b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 00:28:28.906991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:28.906961 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8b3bf0-729qb" Oct 30 00:28:28.994229 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:28.994127 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28c5ba02-cec3-4d4d-b0bf-67c2db6b981d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28c5ba02-cec3-4d4d-b0bf-67c2db6b981d-maas-api-openshift-ca\") pod \"28c5ba02-cec3-4d4d-b0bf-67c2db6b981d\" (UID: \"28c5ba02-cec3-4d4d-b0bf-67c2db6b981d\") " Oct 30 00:28:28.994409 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:28.994257 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28c5ba02-cec3-4d4d-b0bf-67c2db6b981d-kube-api-access-m25vv\" (UniqueName: \"kubernetes.io/projected/28c5ba02-cec3-4d4d-b0bf-67c2db6b981d-kube-api-access-m25vv\") pod \"28c5ba02-cec3-4d4d-b0bf-67c2db6b981d\" (UID: \"28c5ba02-cec3-4d4d-b0bf-67c2db6b981d\") " Oct 30 00:28:28.994409 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:28.994292 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28c5ba02-cec3-4d4d-b0bf-67c2db6b981d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28c5ba02-cec3-4d4d-b0bf-67c2db6b981d-maas-api-xks-ca\") pod \"28c5ba02-cec3-4d4d-b0bf-67c2db6b981d\" (UID: \"28c5ba02-cec3-4d4d-b0bf-67c2db6b981d\") " Oct 30 00:28:28.994977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:28.994944 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28c5ba02-cec3-4d4d-b0bf-67c2db6b981d-maas-api-openshift-ca" pod "28c5ba02-cec3-4d4d-b0bf-67c2db6b981d" (UID: "28c5ba02-cec3-4d4d-b0bf-67c2db6b981d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:28:28.999685 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:28.999635 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28c5ba02-cec3-4d4d-b0bf-67c2db6b981d-maas-api-xks-ca" pod "28c5ba02-cec3-4d4d-b0bf-67c2db6b981d" (UID: "28c5ba02-cec3-4d4d-b0bf-67c2db6b981d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:28:28.999993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:28.999969 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28c5ba02-cec3-4d4d-b0bf-67c2db6b981d-kube-api-access-m25vv" pod "28c5ba02-cec3-4d4d-b0bf-67c2db6b981d" (UID: "28c5ba02-cec3-4d4d-b0bf-67c2db6b981d"). InnerVolumeSpecName "kube-api-access-m25vv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:28:29.096863 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:29.096823 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28c5ba02-cec3-4d4d-b0bf-67c2db6b981d-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:29.096863 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:29.096864 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m25vv\" (UniqueName: \"kubernetes.io/projected/28c5ba02-cec3-4d4d-b0bf-67c2db6b981d-kube-api-access-m25vv\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:29.097146 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:29.096879 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28c5ba02-cec3-4d4d-b0bf-67c2db6b981d-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:29.262617 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:29.262585 5001 generic.go:356] "Generic (PLEG): container finished" podID="d0d7bfe5-4113-4a81-8015-936389553f69" containerID="426f05c55e3ce307838f2da9ff6e97ae53c026192023e0e7d1f26a6dfed3a6f1" exitCode=0 Oct 30 00:28:29.263038 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:29.262678 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bb16a8-hgdpt" event={"ID":"d0d7bfe5-4113-4a81-8015-936389553f69","Type":"ContainerDied","Data":"426f05c55e3ce307838f2da9ff6e97ae53c026192023e0e7d1f26a6dfed3a6f1"} Oct 30 00:28:29.264362 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:29.264341 5001 generic.go:356] "Generic (PLEG): container finished" podID="c0a4294e-62aa-49ea-928a-da94c5381802" containerID="1159211d6593cc026f268183bca70f915ab5e191fb5eaa1f0842fc5643dfd0cc" exitCode=0 Oct 30 00:28:29.264445 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:29.264387 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f8397b-s8hxd" event={"ID":"c0a4294e-62aa-49ea-928a-da94c5381802","Type":"ContainerDied","Data":"1159211d6593cc026f268183bca70f915ab5e191fb5eaa1f0842fc5643dfd0cc"} Oct 30 00:28:29.265784 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:29.265746 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8b3bf0-729qb" event={"ID":"28c5ba02-cec3-4d4d-b0bf-67c2db6b981d","Type":"ContainerDied","Data":"831922d7fc804959e3c0e4842d19d6807c0ba0e0d0f48ccc6d4b7553e8b4e727"} Oct 30 00:28:29.265885 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:29.265786 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8b3bf0-729qb" Oct 30 00:28:29.265885 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:29.265803 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="831922d7fc804959e3c0e4842d19d6807c0ba0e0d0f48ccc6d4b7553e8b4e727" Oct 30 00:28:29.272773 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:29.272757 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:28:29.273781 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:29.273757 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:28:29.421792 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:29.421757 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:28:29.422288 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:29.422264 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:28:29.423090 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:29.423074 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:28:29.681558 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:29.681529 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:28:29.682585 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:29.682564 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:28:30.462294 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:30.462268 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bb16a8-hgdpt" Oct 30 00:28:30.516808 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:30.516771 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f8397b-s8hxd" Oct 30 00:28:30.535054 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:30.534958 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0d7bfe5-4113-4a81-8015-936389553f69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0d7bfe5-4113-4a81-8015-936389553f69-maas-api-xks-ca\") pod \"d0d7bfe5-4113-4a81-8015-936389553f69\" (UID: \"d0d7bfe5-4113-4a81-8015-936389553f69\") " Oct 30 00:28:30.535233 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:30.535168 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0d7bfe5-4113-4a81-8015-936389553f69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0d7bfe5-4113-4a81-8015-936389553f69-maas-api-openshift-ca\") pod \"d0d7bfe5-4113-4a81-8015-936389553f69\" (UID: \"d0d7bfe5-4113-4a81-8015-936389553f69\") " Oct 30 00:28:30.535406 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:30.535385 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0d7bfe5-4113-4a81-8015-936389553f69-kube-api-access-9cvwd\" (UniqueName: \"kubernetes.io/projected/d0d7bfe5-4113-4a81-8015-936389553f69-kube-api-access-9cvwd\") pod \"d0d7bfe5-4113-4a81-8015-936389553f69\" (UID: \"d0d7bfe5-4113-4a81-8015-936389553f69\") " Oct 30 00:28:30.535525 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:30.535469 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0d7bfe5-4113-4a81-8015-936389553f69-maas-api-openshift-ca" pod "d0d7bfe5-4113-4a81-8015-936389553f69" (UID: "d0d7bfe5-4113-4a81-8015-936389553f69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:28:30.536270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:30.536232 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0d7bfe5-4113-4a81-8015-936389553f69-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:30.537619 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:30.537594 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d7bfe5-4113-4a81-8015-936389553f69-kube-api-access-9cvwd" pod "d0d7bfe5-4113-4a81-8015-936389553f69" (UID: "d0d7bfe5-4113-4a81-8015-936389553f69"). InnerVolumeSpecName "kube-api-access-9cvwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:28:30.537721 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:30.537638 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0d7bfe5-4113-4a81-8015-936389553f69-maas-api-xks-ca" pod "d0d7bfe5-4113-4a81-8015-936389553f69" (UID: "d0d7bfe5-4113-4a81-8015-936389553f69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:28:30.637268 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:30.637226 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0a4294e-62aa-49ea-928a-da94c5381802-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a4294e-62aa-49ea-928a-da94c5381802-maas-api-xks-ca\") pod \"c0a4294e-62aa-49ea-928a-da94c5381802\" (UID: \"c0a4294e-62aa-49ea-928a-da94c5381802\") " Oct 30 00:28:30.637457 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:30.637372 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0a4294e-62aa-49ea-928a-da94c5381802-kube-api-access-x76bc\" (UniqueName: \"kubernetes.io/projected/c0a4294e-62aa-49ea-928a-da94c5381802-kube-api-access-x76bc\") pod \"c0a4294e-62aa-49ea-928a-da94c5381802\" (UID: \"c0a4294e-62aa-49ea-928a-da94c5381802\") " Oct 30 00:28:30.637566 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:30.637546 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0a4294e-62aa-49ea-928a-da94c5381802-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a4294e-62aa-49ea-928a-da94c5381802-maas-api-openshift-ca\") pod \"c0a4294e-62aa-49ea-928a-da94c5381802\" (UID: \"c0a4294e-62aa-49ea-928a-da94c5381802\") " Oct 30 00:28:30.637932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:30.637891 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0a4294e-62aa-49ea-928a-da94c5381802-maas-api-openshift-ca" pod "c0a4294e-62aa-49ea-928a-da94c5381802" (UID: "c0a4294e-62aa-49ea-928a-da94c5381802"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:28:30.638315 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:30.638294 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a4294e-62aa-49ea-928a-da94c5381802-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:30.638402 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:30.638317 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cvwd\" (UniqueName: \"kubernetes.io/projected/d0d7bfe5-4113-4a81-8015-936389553f69-kube-api-access-9cvwd\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:30.638402 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:30.638327 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0d7bfe5-4113-4a81-8015-936389553f69-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:30.639493 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:30.639464 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0a4294e-62aa-49ea-928a-da94c5381802-maas-api-xks-ca" pod "c0a4294e-62aa-49ea-928a-da94c5381802" (UID: "c0a4294e-62aa-49ea-928a-da94c5381802"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:28:30.639899 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:30.639879 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0a4294e-62aa-49ea-928a-da94c5381802-kube-api-access-x76bc" pod "c0a4294e-62aa-49ea-928a-da94c5381802" (UID: "c0a4294e-62aa-49ea-928a-da94c5381802"). InnerVolumeSpecName "kube-api-access-x76bc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:28:30.739642 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:30.739612 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a4294e-62aa-49ea-928a-da94c5381802-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:30.739642 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:30.739642 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x76bc\" (UniqueName: \"kubernetes.io/projected/c0a4294e-62aa-49ea-928a-da94c5381802-kube-api-access-x76bc\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:31.009950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:31.009895 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" Oct 30 00:28:31.022977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:31.022949 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" Oct 30 00:28:31.285261 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:31.285232 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bb16a8-hgdpt" Oct 30 00:28:31.285261 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:31.285233 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bb16a8-hgdpt" event={"ID":"d0d7bfe5-4113-4a81-8015-936389553f69","Type":"ContainerDied","Data":"41c7763a04c87fd46b7bdd27514c17d28acf57f0fff96e5a2c0d38b66c5cf8aa"} Oct 30 00:28:31.285579 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:31.285296 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41c7763a04c87fd46b7bdd27514c17d28acf57f0fff96e5a2c0d38b66c5cf8aa" Oct 30 00:28:31.287089 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:31.287061 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f8397b-s8hxd" event={"ID":"c0a4294e-62aa-49ea-928a-da94c5381802","Type":"ContainerDied","Data":"4b415efaf12f942f8bab78b9afc96e6710f6e1a1c22433f5095a3738b34118ab"} Oct 30 00:28:31.287089 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:31.287081 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f8397b-s8hxd" Oct 30 00:28:31.287295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:31.287108 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b415efaf12f942f8bab78b9afc96e6710f6e1a1c22433f5095a3738b34118ab" Oct 30 00:28:31.299414 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:31.299391 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" Oct 30 00:28:31.842481 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:31.842444 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-522ea7-ttr8z"] Oct 30 00:28:31.843307 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:31.843288 5001 cpu_manager.go:414] "containerMap: removing container" podUID="28c5ba02-cec3-4d4d-b0bf-67c2db6b981d" containerName="revoke-keys" Oct 30 00:28:31.843307 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:31.843307 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28c5ba02-cec3-4d4d-b0bf-67c2db6b981d" containerName="revoke-keys" Oct 30 00:28:31.843425 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:31.843330 5001 cpu_manager.go:414] "containerMap: removing container" podUID="c0a4294e-62aa-49ea-928a-da94c5381802" containerName="revoke-keys" Oct 30 00:28:31.843425 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:31.843335 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0a4294e-62aa-49ea-928a-da94c5381802" containerName="revoke-keys" Oct 30 00:28:31.843425 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:31.843355 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d0d7bfe5-4113-4a81-8015-936389553f69" containerName="revoke-keys" Oct 30 00:28:31.843425 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:31.843360 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0d7bfe5-4113-4a81-8015-936389553f69" containerName="revoke-keys" Oct 30 00:28:31.843557 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:31.843471 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="28c5ba02-cec3-4d4d-b0bf-67c2db6b981d" containerName="revoke-keys" Oct 30 00:28:31.843557 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:31.843483 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0a4294e-62aa-49ea-928a-da94c5381802" containerName="revoke-keys" Oct 30 00:28:31.843557 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:31.843492 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0d7bfe5-4113-4a81-8015-936389553f69" containerName="revoke-keys" Oct 30 00:28:31.847563 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:31.847545 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-522ea7-ttr8z" Oct 30 00:28:31.861559 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:31.861529 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:28:31.861559 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:31.861530 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:28:31.861754 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:31.861585 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:28:31.871139 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:31.871118 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:28:31.878340 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:31.878299 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-522ea7-ttr8z"] Oct 30 00:28:31.989071 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:31.989036 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjtpm\" (UniqueName: \"kubernetes.io/projected/ceefd853-1208-4724-aad4-87b83442d3f4-kube-api-access-sjtpm\") pod \"maas-api-revoke-keys-e2e-praxis-a-522ea7-ttr8z\" (UID: \"ceefd853-1208-4724-aad4-87b83442d3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-522ea7-ttr8z" Oct 30 00:28:31.989689 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:31.989648 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceefd853-1208-4724-aad4-87b83442d3f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-522ea7-ttr8z\" (UID: \"ceefd853-1208-4724-aad4-87b83442d3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-522ea7-ttr8z" Oct 30 00:28:31.989836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:31.989717 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceefd853-1208-4724-aad4-87b83442d3f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-522ea7-ttr8z\" (UID: \"ceefd853-1208-4724-aad4-87b83442d3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-522ea7-ttr8z" Oct 30 00:28:32.091392 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:32.091351 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceefd853-1208-4724-aad4-87b83442d3f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-522ea7-ttr8z\" (UID: \"ceefd853-1208-4724-aad4-87b83442d3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-522ea7-ttr8z" Oct 30 00:28:32.091565 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:32.091407 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceefd853-1208-4724-aad4-87b83442d3f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-522ea7-ttr8z\" (UID: \"ceefd853-1208-4724-aad4-87b83442d3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-522ea7-ttr8z" Oct 30 00:28:32.091656 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:32.091572 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjtpm\" (UniqueName: \"kubernetes.io/projected/ceefd853-1208-4724-aad4-87b83442d3f4-kube-api-access-sjtpm\") pod \"maas-api-revoke-keys-e2e-praxis-a-522ea7-ttr8z\" (UID: \"ceefd853-1208-4724-aad4-87b83442d3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-522ea7-ttr8z" Oct 30 00:28:32.092158 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:32.092137 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceefd853-1208-4724-aad4-87b83442d3f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-522ea7-ttr8z\" (UID: \"ceefd853-1208-4724-aad4-87b83442d3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-522ea7-ttr8z" Oct 30 00:28:32.093958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:32.093896 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceefd853-1208-4724-aad4-87b83442d3f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-522ea7-ttr8z\" (UID: \"ceefd853-1208-4724-aad4-87b83442d3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-522ea7-ttr8z" Oct 30 00:28:32.112183 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:32.112156 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjtpm\" (UniqueName: \"kubernetes.io/projected/ceefd853-1208-4724-aad4-87b83442d3f4-kube-api-access-sjtpm\") pod \"maas-api-revoke-keys-e2e-praxis-a-522ea7-ttr8z\" (UID: \"ceefd853-1208-4724-aad4-87b83442d3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-522ea7-ttr8z" Oct 30 00:28:32.163197 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:32.163166 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-522ea7-ttr8z" Oct 30 00:28:32.472265 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:28:32.472225 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceefd853_1208_4724_aad4_87b83442d3f4.slice/crio-ac4496830e655eba82c082122f0e2bd541b253da84867ff99904a63f8f4f7212 WatchSource:0}: Error finding container ac4496830e655eba82c082122f0e2bd541b253da84867ff99904a63f8f4f7212: Status 404 returned error can't find the container with id ac4496830e655eba82c082122f0e2bd541b253da84867ff99904a63f8f4f7212 Oct 30 00:28:32.474299 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:32.474269 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-522ea7-ttr8z"] Oct 30 00:28:33.311335 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:33.311296 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-522ea7-ttr8z" event={"ID":"ceefd853-1208-4724-aad4-87b83442d3f4","Type":"ContainerStarted","Data":"9dd4a7bf9238ef86a2f0459715c4836910a54cfa4f1e60b4395e9fb739250bdd"} Oct 30 00:28:33.311839 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:33.311354 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-522ea7-ttr8z" event={"ID":"ceefd853-1208-4724-aad4-87b83442d3f4","Type":"ContainerStarted","Data":"ac4496830e655eba82c082122f0e2bd541b253da84867ff99904a63f8f4f7212"} Oct 30 00:28:33.348670 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:33.348621 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-522ea7-ttr8z" podStartSLOduration=2.34859704 podStartE2EDuration="2.34859704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:28:33.348228938 +0000 UTC m=+1428.553730118" watchObservedRunningTime="2026-10-30 00:28:33.34859704 +0000 UTC m=+1428.554098217" Oct 30 00:28:34.246485 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:34.246444 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8b3bf0-729qb"] Oct 30 00:28:34.251109 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:34.251076 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8b3bf0-729qb"] Oct 30 00:28:34.332064 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:34.332018 5001 generic.go:356] "Generic (PLEG): container finished" podID="ceefd853-1208-4724-aad4-87b83442d3f4" containerID="9dd4a7bf9238ef86a2f0459715c4836910a54cfa4f1e60b4395e9fb739250bdd" exitCode=0 Oct 30 00:28:34.332650 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:34.332232 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-522ea7-ttr8z" event={"ID":"ceefd853-1208-4724-aad4-87b83442d3f4","Type":"ContainerDied","Data":"9dd4a7bf9238ef86a2f0459715c4836910a54cfa4f1e60b4395e9fb739250bdd"} Oct 30 00:28:34.922427 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:34.922389 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28c5ba02-cec3-4d4d-b0bf-67c2db6b981d" path="/var/lib/kubelet/pods/28c5ba02-cec3-4d4d-b0bf-67c2db6b981d/volumes" Oct 30 00:28:35.517750 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:35.517721 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-522ea7-ttr8z" Oct 30 00:28:35.592473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:35.587078 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ceefd853-1208-4724-aad4-87b83442d3f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceefd853-1208-4724-aad4-87b83442d3f4-maas-api-xks-ca\") pod \"ceefd853-1208-4724-aad4-87b83442d3f4\" (UID: \"ceefd853-1208-4724-aad4-87b83442d3f4\") " Oct 30 00:28:35.592473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:35.587355 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceefd853-1208-4724-aad4-87b83442d3f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceefd853-1208-4724-aad4-87b83442d3f4-maas-api-openshift-ca\") pod \"ceefd853-1208-4724-aad4-87b83442d3f4\" (UID: \"ceefd853-1208-4724-aad4-87b83442d3f4\") " Oct 30 00:28:35.592473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:35.587460 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceefd853-1208-4724-aad4-87b83442d3f4-kube-api-access-sjtpm\" (UniqueName: \"kubernetes.io/projected/ceefd853-1208-4724-aad4-87b83442d3f4-kube-api-access-sjtpm\") pod \"ceefd853-1208-4724-aad4-87b83442d3f4\" (UID: \"ceefd853-1208-4724-aad4-87b83442d3f4\") " Oct 30 00:28:35.592473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:35.591161 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceefd853-1208-4724-aad4-87b83442d3f4-maas-api-openshift-ca" pod "ceefd853-1208-4724-aad4-87b83442d3f4" (UID: "ceefd853-1208-4724-aad4-87b83442d3f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:28:35.595869 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:35.594026 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceefd853-1208-4724-aad4-87b83442d3f4-kube-api-access-sjtpm" pod "ceefd853-1208-4724-aad4-87b83442d3f4" (UID: "ceefd853-1208-4724-aad4-87b83442d3f4"). InnerVolumeSpecName "kube-api-access-sjtpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:28:35.600970 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:35.597416 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceefd853-1208-4724-aad4-87b83442d3f4-maas-api-xks-ca" pod "ceefd853-1208-4724-aad4-87b83442d3f4" (UID: "ceefd853-1208-4724-aad4-87b83442d3f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:28:35.692344 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:35.691407 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjtpm\" (UniqueName: \"kubernetes.io/projected/ceefd853-1208-4724-aad4-87b83442d3f4-kube-api-access-sjtpm\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:35.692344 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:35.691452 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceefd853-1208-4724-aad4-87b83442d3f4-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:35.692344 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:35.691468 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceefd853-1208-4724-aad4-87b83442d3f4-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:36.462557 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:36.462521 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-522ea7-ttr8z" event={"ID":"ceefd853-1208-4724-aad4-87b83442d3f4","Type":"ContainerDied","Data":"ac4496830e655eba82c082122f0e2bd541b253da84867ff99904a63f8f4f7212"} Oct 30 00:28:36.462820 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:36.462799 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac4496830e655eba82c082122f0e2bd541b253da84867ff99904a63f8f4f7212" Oct 30 00:28:36.463181 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:36.463164 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-522ea7-ttr8z" Oct 30 00:28:41.356576 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:41.356540 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m"] Oct 30 00:28:41.357123 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:41.356825 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" containerID="cri-o://cfd0797bc44839ccf8a875a3aeb6b7c395345c4fc65808485ff08bfd7c7b00d7" gracePeriod=30 Oct 30 00:28:41.927352 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:41.927317 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-00df5f-jjgpb"] Oct 30 00:28:41.928155 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:41.928126 5001 cpu_manager.go:414] "containerMap: removing container" podUID="ceefd853-1208-4724-aad4-87b83442d3f4" containerName="revoke-keys" Oct 30 00:28:41.928155 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:41.928149 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceefd853-1208-4724-aad4-87b83442d3f4" containerName="revoke-keys" Oct 30 00:28:41.928331 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:41.928319 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceefd853-1208-4724-aad4-87b83442d3f4" containerName="revoke-keys" Oct 30 00:28:41.932243 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:41.932227 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-00df5f-jjgpb" Oct 30 00:28:41.951970 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:41.951947 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-00df5f-jjgpb"] Oct 30 00:28:41.955698 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:41.955679 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:28:41.955698 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:41.955679 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:28:41.983557 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:41.983525 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5046332-5ea3-4746-8100-f28a12a95aee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-00df5f-jjgpb\" (UID: \"b5046332-5ea3-4746-8100-f28a12a95aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-00df5f-jjgpb" Oct 30 00:28:41.983676 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:41.983616 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5046332-5ea3-4746-8100-f28a12a95aee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-00df5f-jjgpb\" (UID: \"b5046332-5ea3-4746-8100-f28a12a95aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-00df5f-jjgpb" Oct 30 00:28:41.983676 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:41.983663 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgt67\" (UniqueName: \"kubernetes.io/projected/b5046332-5ea3-4746-8100-f28a12a95aee-kube-api-access-wgt67\") pod \"maas-api-revoke-keys-e2e-praxis-b-00df5f-jjgpb\" (UID: \"b5046332-5ea3-4746-8100-f28a12a95aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-00df5f-jjgpb" Oct 30 00:28:41.986611 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:41.986588 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:28:41.986693 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:41.986621 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:28:42.085141 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:42.085106 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgt67\" (UniqueName: \"kubernetes.io/projected/b5046332-5ea3-4746-8100-f28a12a95aee-kube-api-access-wgt67\") pod \"maas-api-revoke-keys-e2e-praxis-b-00df5f-jjgpb\" (UID: \"b5046332-5ea3-4746-8100-f28a12a95aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-00df5f-jjgpb" Oct 30 00:28:42.085454 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:42.085431 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5046332-5ea3-4746-8100-f28a12a95aee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-00df5f-jjgpb\" (UID: \"b5046332-5ea3-4746-8100-f28a12a95aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-00df5f-jjgpb" Oct 30 00:28:42.085542 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:42.085473 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5046332-5ea3-4746-8100-f28a12a95aee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-00df5f-jjgpb\" (UID: \"b5046332-5ea3-4746-8100-f28a12a95aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-00df5f-jjgpb" Oct 30 00:28:42.086205 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:42.086189 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5046332-5ea3-4746-8100-f28a12a95aee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-00df5f-jjgpb\" (UID: \"b5046332-5ea3-4746-8100-f28a12a95aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-00df5f-jjgpb" Oct 30 00:28:42.087894 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:42.087870 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5046332-5ea3-4746-8100-f28a12a95aee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-00df5f-jjgpb\" (UID: \"b5046332-5ea3-4746-8100-f28a12a95aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-00df5f-jjgpb" Oct 30 00:28:42.105775 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:42.105752 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgt67\" (UniqueName: \"kubernetes.io/projected/b5046332-5ea3-4746-8100-f28a12a95aee-kube-api-access-wgt67\") pod \"maas-api-revoke-keys-e2e-praxis-b-00df5f-jjgpb\" (UID: \"b5046332-5ea3-4746-8100-f28a12a95aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-00df5f-jjgpb" Oct 30 00:28:42.247195 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:42.247118 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-00df5f-jjgpb" Oct 30 00:28:42.551281 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:42.548865 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-522ea7-7dcbbd964-4x4lf"] Oct 30 00:28:42.576507 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:42.576048 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-522ea7-7dfd54bbf5-2frb5"] Oct 30 00:28:42.589957 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:42.588131 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-00df5f-jjgpb"] Oct 30 00:28:42.809026 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:42.808629 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-00df5f-jjgpb" event={"ID":"b5046332-5ea3-4746-8100-f28a12a95aee","Type":"ContainerStarted","Data":"f695972c045b083829446bb150eab94e131b6afb8d14ec927884cfb915e377a4"} Oct 30 00:28:43.891457 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:43.891419 5001 generic.go:356] "Generic (PLEG): container finished" podID="b5046332-5ea3-4746-8100-f28a12a95aee" containerID="542329919c01f374ea9b12c20ad25bb165bb182f4727c28beba3cfce88b7fde0" exitCode=0 Oct 30 00:28:43.892127 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:43.892099 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-00df5f-jjgpb" event={"ID":"b5046332-5ea3-4746-8100-f28a12a95aee","Type":"ContainerDied","Data":"542329919c01f374ea9b12c20ad25bb165bb182f4727c28beba3cfce88b7fde0"} Oct 30 00:28:44.452934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:44.452337 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-522ea7-ttr8z"] Oct 30 00:28:44.461329 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:44.461293 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-522ea7-ttr8z"] Oct 30 00:28:45.066707 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:45.066671 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:28:45.144977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:45.077035 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:28:45.225086 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:45.225045 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceefd853-1208-4724-aad4-87b83442d3f4" path="/var/lib/kubelet/pods/ceefd853-1208-4724-aad4-87b83442d3f4/volumes" Oct 30 00:28:45.705144 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:45.705116 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-00df5f-jjgpb" Oct 30 00:28:45.872659 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:45.872559 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5046332-5ea3-4746-8100-f28a12a95aee-kube-api-access-wgt67\" (UniqueName: \"kubernetes.io/projected/b5046332-5ea3-4746-8100-f28a12a95aee-kube-api-access-wgt67\") pod \"b5046332-5ea3-4746-8100-f28a12a95aee\" (UID: \"b5046332-5ea3-4746-8100-f28a12a95aee\") " Oct 30 00:28:45.872902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:45.872880 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5046332-5ea3-4746-8100-f28a12a95aee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5046332-5ea3-4746-8100-f28a12a95aee-maas-api-openshift-ca\") pod \"b5046332-5ea3-4746-8100-f28a12a95aee\" (UID: \"b5046332-5ea3-4746-8100-f28a12a95aee\") " Oct 30 00:28:45.873067 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:45.872978 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5046332-5ea3-4746-8100-f28a12a95aee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5046332-5ea3-4746-8100-f28a12a95aee-maas-api-xks-ca\") pod \"b5046332-5ea3-4746-8100-f28a12a95aee\" (UID: \"b5046332-5ea3-4746-8100-f28a12a95aee\") " Oct 30 00:28:45.874292 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:45.874099 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5046332-5ea3-4746-8100-f28a12a95aee-maas-api-openshift-ca" pod "b5046332-5ea3-4746-8100-f28a12a95aee" (UID: "b5046332-5ea3-4746-8100-f28a12a95aee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:28:45.876757 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:45.876154 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5046332-5ea3-4746-8100-f28a12a95aee-maas-api-xks-ca" pod "b5046332-5ea3-4746-8100-f28a12a95aee" (UID: "b5046332-5ea3-4746-8100-f28a12a95aee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:28:45.876757 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:45.876480 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5046332-5ea3-4746-8100-f28a12a95aee-kube-api-access-wgt67" pod "b5046332-5ea3-4746-8100-f28a12a95aee" (UID: "b5046332-5ea3-4746-8100-f28a12a95aee"). InnerVolumeSpecName "kube-api-access-wgt67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:28:45.975733 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:45.975692 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgt67\" (UniqueName: \"kubernetes.io/projected/b5046332-5ea3-4746-8100-f28a12a95aee-kube-api-access-wgt67\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:45.975733 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:45.975733 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5046332-5ea3-4746-8100-f28a12a95aee-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:45.975988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:45.975749 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5046332-5ea3-4746-8100-f28a12a95aee-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:46.101487 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:46.101458 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-00df5f-jjgpb" Oct 30 00:28:46.102008 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:46.101447 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-00df5f-jjgpb" event={"ID":"b5046332-5ea3-4746-8100-f28a12a95aee","Type":"ContainerDied","Data":"f695972c045b083829446bb150eab94e131b6afb8d14ec927884cfb915e377a4"} Oct 30 00:28:46.102008 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:46.101636 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f695972c045b083829446bb150eab94e131b6afb8d14ec927884cfb915e377a4" Oct 30 00:28:46.826953 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:46.816631 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:28:46.906148 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:46.906108 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-istio-envoy\") pod \"e34c0d4a-0def-4964-8c2a-4599941a3a03\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " Oct 30 00:28:46.906346 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:46.906282 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-workload-socket\") pod \"e34c0d4a-0def-4964-8c2a-4599941a3a03\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " Oct 30 00:28:46.906601 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:46.906556 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-workload-socket" pod "e34c0d4a-0def-4964-8c2a-4599941a3a03" (UID: "e34c0d4a-0def-4964-8c2a-4599941a3a03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:28:46.906769 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:46.906686 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-workload-certs" pod "e34c0d4a-0def-4964-8c2a-4599941a3a03" (UID: "e34c0d4a-0def-4964-8c2a-4599941a3a03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:28:46.906866 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:46.906847 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-workload-certs\") pod \"e34c0d4a-0def-4964-8c2a-4599941a3a03\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " Oct 30 00:28:46.907009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:46.906992 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e34c0d4a-0def-4964-8c2a-4599941a3a03-kube-api-access-d4w5b\" (UniqueName: \"kubernetes.io/projected/e34c0d4a-0def-4964-8c2a-4599941a3a03-kube-api-access-d4w5b\") pod \"e34c0d4a-0def-4964-8c2a-4599941a3a03\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " Oct 30 00:28:46.907155 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:46.907140 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e34c0d4a-0def-4964-8c2a-4599941a3a03-istio-token\" (UniqueName: \"kubernetes.io/projected/e34c0d4a-0def-4964-8c2a-4599941a3a03-istio-token\") pod \"e34c0d4a-0def-4964-8c2a-4599941a3a03\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " Oct 30 00:28:46.907276 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:46.907261 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-istio-data\") pod \"e34c0d4a-0def-4964-8c2a-4599941a3a03\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " Oct 30 00:28:46.907433 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:46.907418 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e34c0d4a-0def-4964-8c2a-4599941a3a03-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e34c0d4a-0def-4964-8c2a-4599941a3a03-istio-podinfo\") pod \"e34c0d4a-0def-4964-8c2a-4599941a3a03\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " Oct 30 00:28:46.907635 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:46.907621 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-credential-socket\") pod \"e34c0d4a-0def-4964-8c2a-4599941a3a03\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " Oct 30 00:28:46.907821 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:46.907807 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e34c0d4a-0def-4964-8c2a-4599941a3a03-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e34c0d4a-0def-4964-8c2a-4599941a3a03-istiod-ca-cert\") pod \"e34c0d4a-0def-4964-8c2a-4599941a3a03\" (UID: \"e34c0d4a-0def-4964-8c2a-4599941a3a03\") " Oct 30 00:28:46.909735 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:46.909707 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:46.909869 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:46.909854 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:46.931790 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:46.914016 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-istio-envoy" pod "e34c0d4a-0def-4964-8c2a-4599941a3a03" (UID: "e34c0d4a-0def-4964-8c2a-4599941a3a03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:28:46.931790 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:46.922109 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-credential-socket" pod "e34c0d4a-0def-4964-8c2a-4599941a3a03" (UID: "e34c0d4a-0def-4964-8c2a-4599941a3a03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:28:46.931790 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:46.922259 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-istio-data" pod "e34c0d4a-0def-4964-8c2a-4599941a3a03" (UID: "e34c0d4a-0def-4964-8c2a-4599941a3a03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:28:46.945677 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:46.933317 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e34c0d4a-0def-4964-8c2a-4599941a3a03-kube-api-access-d4w5b" pod "e34c0d4a-0def-4964-8c2a-4599941a3a03" (UID: "e34c0d4a-0def-4964-8c2a-4599941a3a03"). InnerVolumeSpecName "kube-api-access-d4w5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:28:46.945677 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:46.933705 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e34c0d4a-0def-4964-8c2a-4599941a3a03-istio-token" pod "e34c0d4a-0def-4964-8c2a-4599941a3a03" (UID: "e34c0d4a-0def-4964-8c2a-4599941a3a03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:28:46.947015 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:46.946205 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e34c0d4a-0def-4964-8c2a-4599941a3a03-istiod-ca-cert" pod "e34c0d4a-0def-4964-8c2a-4599941a3a03" (UID: "e34c0d4a-0def-4964-8c2a-4599941a3a03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:28:46.950171 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:46.950144 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e34c0d4a-0def-4964-8c2a-4599941a3a03-istio-podinfo" pod "e34c0d4a-0def-4964-8c2a-4599941a3a03" (UID: "e34c0d4a-0def-4964-8c2a-4599941a3a03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:28:47.029993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:47.014324 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:47.029993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:47.014368 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e34c0d4a-0def-4964-8c2a-4599941a3a03-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:47.029993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:47.014387 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:47.029993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:47.014401 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4w5b\" (UniqueName: \"kubernetes.io/projected/e34c0d4a-0def-4964-8c2a-4599941a3a03-kube-api-access-d4w5b\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:47.029993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:47.014417 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e34c0d4a-0def-4964-8c2a-4599941a3a03-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:47.029993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:47.014432 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e34c0d4a-0def-4964-8c2a-4599941a3a03-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:47.029993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:47.014444 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e34c0d4a-0def-4964-8c2a-4599941a3a03-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:47.133936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:47.133158 5001 generic.go:356] "Generic (PLEG): container finished" podID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerID="cfd0797bc44839ccf8a875a3aeb6b7c395345c4fc65808485ff08bfd7c7b00d7" exitCode=0 Oct 30 00:28:47.133936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:47.133482 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" event={"ID":"e34c0d4a-0def-4964-8c2a-4599941a3a03","Type":"ContainerDied","Data":"cfd0797bc44839ccf8a875a3aeb6b7c395345c4fc65808485ff08bfd7c7b00d7"} Oct 30 00:28:47.133936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:47.133533 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" event={"ID":"e34c0d4a-0def-4964-8c2a-4599941a3a03","Type":"ContainerDied","Data":"94d87f46b63530846c1ec019e04a61561fe762b6d21c4605ea64af8a4b36a10b"} Oct 30 00:28:47.133936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:47.133564 5001 scope.go:122] "RemoveContainer" containerID="cfd0797bc44839ccf8a875a3aeb6b7c395345c4fc65808485ff08bfd7c7b00d7" Oct 30 00:28:47.133936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:47.133745 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m" Oct 30 00:28:47.151796 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:47.151769 5001 scope.go:122] "RemoveContainer" containerID="6171c8650a197691769b8e0b1b0674f8651df0f15863762f9b90d8d9ec49c156" Oct 30 00:28:47.162894 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:47.162875 5001 scope.go:122] "RemoveContainer" containerID="cfd0797bc44839ccf8a875a3aeb6b7c395345c4fc65808485ff08bfd7c7b00d7" Oct 30 00:28:47.163196 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:28:47.163176 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfd0797bc44839ccf8a875a3aeb6b7c395345c4fc65808485ff08bfd7c7b00d7\": container with ID starting with cfd0797bc44839ccf8a875a3aeb6b7c395345c4fc65808485ff08bfd7c7b00d7 not found: ID does not exist" containerID="cfd0797bc44839ccf8a875a3aeb6b7c395345c4fc65808485ff08bfd7c7b00d7" Oct 30 00:28:47.163262 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:47.163205 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfd0797bc44839ccf8a875a3aeb6b7c395345c4fc65808485ff08bfd7c7b00d7"} err="failed to get container status \"cfd0797bc44839ccf8a875a3aeb6b7c395345c4fc65808485ff08bfd7c7b00d7\": rpc error: code = NotFound desc = could not find container \"cfd0797bc44839ccf8a875a3aeb6b7c395345c4fc65808485ff08bfd7c7b00d7\": container with ID starting with cfd0797bc44839ccf8a875a3aeb6b7c395345c4fc65808485ff08bfd7c7b00d7 not found: ID does not exist" Oct 30 00:28:47.163262 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:47.163221 5001 scope.go:122] "RemoveContainer" containerID="6171c8650a197691769b8e0b1b0674f8651df0f15863762f9b90d8d9ec49c156" Oct 30 00:28:47.163477 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:28:47.163459 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6171c8650a197691769b8e0b1b0674f8651df0f15863762f9b90d8d9ec49c156\": container with ID starting with 6171c8650a197691769b8e0b1b0674f8651df0f15863762f9b90d8d9ec49c156 not found: ID does not exist" containerID="6171c8650a197691769b8e0b1b0674f8651df0f15863762f9b90d8d9ec49c156" Oct 30 00:28:47.163523 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:47.163485 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6171c8650a197691769b8e0b1b0674f8651df0f15863762f9b90d8d9ec49c156"} err="failed to get container status \"6171c8650a197691769b8e0b1b0674f8651df0f15863762f9b90d8d9ec49c156\": rpc error: code = NotFound desc = could not find container \"6171c8650a197691769b8e0b1b0674f8651df0f15863762f9b90d8d9ec49c156\": container with ID starting with 6171c8650a197691769b8e0b1b0674f8651df0f15863762f9b90d8d9ec49c156 not found: ID does not exist" Oct 30 00:28:47.184284 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:47.184255 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m"] Oct 30 00:28:47.190540 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:47.190514 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-8b3bf0-openshift-default-76576665cc-9rp9m"] Oct 30 00:28:47.346672 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:28:47.346584 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:28:47.550029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:47.549989 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-522ea7-7dcbbd964-4x4lf" podUID="3c40ae47-fba8-4f1a-a26e-d8ddcfb636af" containerName="payload-processing" containerID="cri-o://a7cd42cbd01058795f3409dedc86e931ace97eb08e693c19ff82618f5dfdab1f" gracePeriod=25 Oct 30 00:28:47.576859 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:47.576824 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-522ea7-7dfd54bbf5-2frb5" podUID="794f4878-3fd8-4f9c-8167-98b6990270cf" containerName="payload-pre-processing" containerID="cri-o://5f5043770887052e1a06ea5ea8d9be0673de3b269edcb88a61671490126a47d8" gracePeriod=25 Oct 30 00:28:48.140978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:48.134154 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-522ea7-7dcbbd964-4x4lf" Oct 30 00:28:48.154630 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:48.154604 5001 generic.go:356] "Generic (PLEG): container finished" podID="3c40ae47-fba8-4f1a-a26e-d8ddcfb636af" containerID="a7cd42cbd01058795f3409dedc86e931ace97eb08e693c19ff82618f5dfdab1f" exitCode=0 Oct 30 00:28:48.154882 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:48.154860 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-522ea7-7dcbbd964-4x4lf" Oct 30 00:28:48.155089 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:48.154773 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-522ea7-7dcbbd964-4x4lf" event={"ID":"3c40ae47-fba8-4f1a-a26e-d8ddcfb636af","Type":"ContainerDied","Data":"a7cd42cbd01058795f3409dedc86e931ace97eb08e693c19ff82618f5dfdab1f"} Oct 30 00:28:48.155227 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:48.155123 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-522ea7-7dcbbd964-4x4lf" event={"ID":"3c40ae47-fba8-4f1a-a26e-d8ddcfb636af","Type":"ContainerDied","Data":"565c1436c6a11accb51a33f30c133abe80fc2534eaf83ec3afa472d4596ba153"} Oct 30 00:28:48.155227 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:48.155149 5001 scope.go:122] "RemoveContainer" containerID="a7cd42cbd01058795f3409dedc86e931ace97eb08e693c19ff82618f5dfdab1f" Oct 30 00:28:48.159040 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:48.158982 5001 generic.go:356] "Generic (PLEG): container finished" podID="794f4878-3fd8-4f9c-8167-98b6990270cf" containerID="5f5043770887052e1a06ea5ea8d9be0673de3b269edcb88a61671490126a47d8" exitCode=0 Oct 30 00:28:48.159040 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:48.159013 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-522ea7-7dfd54bbf5-2frb5" event={"ID":"794f4878-3fd8-4f9c-8167-98b6990270cf","Type":"ContainerDied","Data":"5f5043770887052e1a06ea5ea8d9be0673de3b269edcb88a61671490126a47d8"} Oct 30 00:28:48.167859 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:48.167838 5001 scope.go:122] "RemoveContainer" containerID="a7cd42cbd01058795f3409dedc86e931ace97eb08e693c19ff82618f5dfdab1f" Oct 30 00:28:48.168219 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:28:48.168200 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7cd42cbd01058795f3409dedc86e931ace97eb08e693c19ff82618f5dfdab1f\": container with ID starting with a7cd42cbd01058795f3409dedc86e931ace97eb08e693c19ff82618f5dfdab1f not found: ID does not exist" containerID="a7cd42cbd01058795f3409dedc86e931ace97eb08e693c19ff82618f5dfdab1f" Oct 30 00:28:48.168308 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:48.168228 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7cd42cbd01058795f3409dedc86e931ace97eb08e693c19ff82618f5dfdab1f"} err="failed to get container status \"a7cd42cbd01058795f3409dedc86e931ace97eb08e693c19ff82618f5dfdab1f\": rpc error: code = NotFound desc = could not find container \"a7cd42cbd01058795f3409dedc86e931ace97eb08e693c19ff82618f5dfdab1f\": container with ID starting with a7cd42cbd01058795f3409dedc86e931ace97eb08e693c19ff82618f5dfdab1f not found: ID does not exist" Oct 30 00:28:48.207562 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:48.207537 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c40ae47-fba8-4f1a-a26e-d8ddcfb636af-kube-api-access-bkkng\" (UniqueName: \"kubernetes.io/projected/3c40ae47-fba8-4f1a-a26e-d8ddcfb636af-kube-api-access-bkkng\") pod \"3c40ae47-fba8-4f1a-a26e-d8ddcfb636af\" (UID: \"3c40ae47-fba8-4f1a-a26e-d8ddcfb636af\") " Oct 30 00:28:48.235103 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:48.208003 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c40ae47-fba8-4f1a-a26e-d8ddcfb636af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c40ae47-fba8-4f1a-a26e-d8ddcfb636af-plugins-config-volume\") pod \"3c40ae47-fba8-4f1a-a26e-d8ddcfb636af\" (UID: \"3c40ae47-fba8-4f1a-a26e-d8ddcfb636af\") " Oct 30 00:28:48.247625 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:48.246558 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c40ae47-fba8-4f1a-a26e-d8ddcfb636af-plugins-config-volume" pod "3c40ae47-fba8-4f1a-a26e-d8ddcfb636af" (UID: "3c40ae47-fba8-4f1a-a26e-d8ddcfb636af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:28:48.248095 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:48.248059 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c40ae47-fba8-4f1a-a26e-d8ddcfb636af-kube-api-access-bkkng" pod "3c40ae47-fba8-4f1a-a26e-d8ddcfb636af" (UID: "3c40ae47-fba8-4f1a-a26e-d8ddcfb636af"). InnerVolumeSpecName "kube-api-access-bkkng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:28:48.258843 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:48.258814 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-522ea7-7dfd54bbf5-2frb5" Oct 30 00:28:48.319937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:48.309862 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/794f4878-3fd8-4f9c-8167-98b6990270cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/794f4878-3fd8-4f9c-8167-98b6990270cf-plugins-config-volume\") pod \"794f4878-3fd8-4f9c-8167-98b6990270cf\" (UID: \"794f4878-3fd8-4f9c-8167-98b6990270cf\") " Oct 30 00:28:48.319937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:48.309952 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/794f4878-3fd8-4f9c-8167-98b6990270cf-kube-api-access-kv46s\" (UniqueName: \"kubernetes.io/projected/794f4878-3fd8-4f9c-8167-98b6990270cf-kube-api-access-kv46s\") pod \"794f4878-3fd8-4f9c-8167-98b6990270cf\" (UID: \"794f4878-3fd8-4f9c-8167-98b6990270cf\") " Oct 30 00:28:48.319937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:48.311548 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c40ae47-fba8-4f1a-a26e-d8ddcfb636af-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:48.319937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:48.311578 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkkng\" (UniqueName: \"kubernetes.io/projected/3c40ae47-fba8-4f1a-a26e-d8ddcfb636af-kube-api-access-bkkng\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:48.319937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:48.314554 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/794f4878-3fd8-4f9c-8167-98b6990270cf-kube-api-access-kv46s" pod "794f4878-3fd8-4f9c-8167-98b6990270cf" (UID: "794f4878-3fd8-4f9c-8167-98b6990270cf"). InnerVolumeSpecName "kube-api-access-kv46s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:28:48.319937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:48.314952 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/794f4878-3fd8-4f9c-8167-98b6990270cf-plugins-config-volume" pod "794f4878-3fd8-4f9c-8167-98b6990270cf" (UID: "794f4878-3fd8-4f9c-8167-98b6990270cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:28:48.433002 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:48.414061 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/794f4878-3fd8-4f9c-8167-98b6990270cf-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:48.433002 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:48.414094 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv46s\" (UniqueName: \"kubernetes.io/projected/794f4878-3fd8-4f9c-8167-98b6990270cf-kube-api-access-kv46s\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:48.498833 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:48.498800 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-522ea7-7dcbbd964-4x4lf"] Oct 30 00:28:48.505008 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:48.504982 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-522ea7-7dcbbd964-4x4lf"] Oct 30 00:28:48.942901 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:48.942870 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c40ae47-fba8-4f1a-a26e-d8ddcfb636af" path="/var/lib/kubelet/pods/3c40ae47-fba8-4f1a-a26e-d8ddcfb636af/volumes" Oct 30 00:28:48.943432 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:48.943416 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" path="/var/lib/kubelet/pods/e34c0d4a-0def-4964-8c2a-4599941a3a03/volumes" Oct 30 00:28:49.318936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:49.308229 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-522ea7-7dfd54bbf5-2frb5" event={"ID":"794f4878-3fd8-4f9c-8167-98b6990270cf","Type":"ContainerDied","Data":"286811a3f27eb1c330defe5f726c55b3aad67bff1b4ea052a3538d57ec09b41e"} Oct 30 00:28:49.318936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:49.308401 5001 scope.go:122] "RemoveContainer" containerID="5f5043770887052e1a06ea5ea8d9be0673de3b269edcb88a61671490126a47d8" Oct 30 00:28:49.318936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:49.308697 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-522ea7-7dfd54bbf5-2frb5" Oct 30 00:28:49.384258 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:49.384230 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-522ea7-7dfd54bbf5-2frb5"] Oct 30 00:28:49.393978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:49.393952 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-522ea7-7dfd54bbf5-2frb5"] Oct 30 00:28:50.938125 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:50.938080 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="794f4878-3fd8-4f9c-8167-98b6990270cf" path="/var/lib/kubelet/pods/794f4878-3fd8-4f9c-8167-98b6990270cf/volumes" Oct 30 00:28:51.599879 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:51.599844 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8"] Oct 30 00:28:51.600434 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:51.600146 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" podUID="4fb15e40-3233-4329-917c-6bfcf33cb894" containerName="istio-proxy" containerID="cri-o://6d6b3c1b010268cd670524f9ab127b3501e621b4281a1f0a033aac64660208e5" gracePeriod=30 Oct 30 00:28:51.937080 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:51.936990 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-bf5958687-wqrzq"] Oct 30 00:28:51.937254 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:51.937221 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-bf5958687-wqrzq" podUID="1359a6e8-77c9-4e05-b11e-27ba0c4573fe" containerName="payload-pre-processing" containerID="cri-o://347f96f80d12786d86391924e2b1e64b378ff8be8741652e052737c888f119a2" gracePeriod=30 Oct 30 00:28:52.792993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.788890 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f8397b-s8hxd"] Oct 30 00:28:52.875433 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.875392 5001 generic.go:356] "Generic (PLEG): container finished" podID="1359a6e8-77c9-4e05-b11e-27ba0c4573fe" containerID="347f96f80d12786d86391924e2b1e64b378ff8be8741652e052737c888f119a2" exitCode=0 Oct 30 00:28:52.875829 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.875802 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-bf5958687-wqrzq" event={"ID":"1359a6e8-77c9-4e05-b11e-27ba0c4573fe","Type":"ContainerDied","Data":"347f96f80d12786d86391924e2b1e64b378ff8be8741652e052737c888f119a2"} Oct 30 00:28:52.883221 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.883196 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-bf5958687-wqrzq" Oct 30 00:28:52.897534 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.893629 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f8397b-s8hxd"] Oct 30 00:28:52.904352 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.899711 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1359a6e8-77c9-4e05-b11e-27ba0c4573fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1359a6e8-77c9-4e05-b11e-27ba0c4573fe-plugins-config-volume\") pod \"1359a6e8-77c9-4e05-b11e-27ba0c4573fe\" (UID: \"1359a6e8-77c9-4e05-b11e-27ba0c4573fe\") " Oct 30 00:28:52.904352 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.899785 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1359a6e8-77c9-4e05-b11e-27ba0c4573fe-kube-api-access-cmz2p\" (UniqueName: \"kubernetes.io/projected/1359a6e8-77c9-4e05-b11e-27ba0c4573fe-kube-api-access-cmz2p\") pod \"1359a6e8-77c9-4e05-b11e-27ba0c4573fe\" (UID: \"1359a6e8-77c9-4e05-b11e-27ba0c4573fe\") " Oct 30 00:28:52.913591 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.913569 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897"] Oct 30 00:28:52.914877 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.914852 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1359a6e8-77c9-4e05-b11e-27ba0c4573fe" containerName="payload-pre-processing" Oct 30 00:28:52.915023 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.915007 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1359a6e8-77c9-4e05-b11e-27ba0c4573fe" containerName="payload-pre-processing" Oct 30 00:28:52.915137 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.915124 5001 cpu_manager.go:414] "containerMap: removing container" podUID="794f4878-3fd8-4f9c-8167-98b6990270cf" containerName="payload-pre-processing" Oct 30 00:28:52.915237 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.915224 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="794f4878-3fd8-4f9c-8167-98b6990270cf" containerName="payload-pre-processing" Oct 30 00:28:52.915358 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.915345 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" Oct 30 00:28:52.915449 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.915438 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" Oct 30 00:28:52.915554 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.915542 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b5046332-5ea3-4746-8100-f28a12a95aee" containerName="revoke-keys" Oct 30 00:28:52.915634 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.915623 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5046332-5ea3-4746-8100-f28a12a95aee" containerName="revoke-keys" Oct 30 00:28:52.915728 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.915717 5001 cpu_manager.go:414] "containerMap: removing container" podUID="3c40ae47-fba8-4f1a-a26e-d8ddcfb636af" containerName="payload-processing" Oct 30 00:28:52.915817 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.915806 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c40ae47-fba8-4f1a-a26e-d8ddcfb636af" containerName="payload-processing" Oct 30 00:28:52.916119 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.916081 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" Oct 30 00:28:52.916374 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.916338 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" Oct 30 00:28:52.916568 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.916534 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1359a6e8-77c9-4e05-b11e-27ba0c4573fe-kube-api-access-cmz2p" pod "1359a6e8-77c9-4e05-b11e-27ba0c4573fe" (UID: "1359a6e8-77c9-4e05-b11e-27ba0c4573fe"). InnerVolumeSpecName "kube-api-access-cmz2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:28:52.916766 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.916748 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5046332-5ea3-4746-8100-f28a12a95aee" containerName="revoke-keys" Oct 30 00:28:52.916876 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.916863 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c40ae47-fba8-4f1a-a26e-d8ddcfb636af" containerName="payload-processing" Oct 30 00:28:52.917007 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.916994 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" Oct 30 00:28:52.917120 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.917107 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1359a6e8-77c9-4e05-b11e-27ba0c4573fe" containerName="payload-pre-processing" Oct 30 00:28:52.917212 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.917201 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="794f4878-3fd8-4f9c-8167-98b6990270cf" containerName="payload-pre-processing" Oct 30 00:28:52.917304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.917293 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e34c0d4a-0def-4964-8c2a-4599941a3a03" containerName="istio-proxy" Oct 30 00:28:52.920650 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.916990 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1359a6e8-77c9-4e05-b11e-27ba0c4573fe-plugins-config-volume" pod "1359a6e8-77c9-4e05-b11e-27ba0c4573fe" (UID: "1359a6e8-77c9-4e05-b11e-27ba0c4573fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:28:52.931788 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.930734 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897"] Oct 30 00:28:52.933191 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.932074 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" Oct 30 00:28:52.946231 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.946206 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-93016b\"/\"default-dockercfg-hjgd6\"" Oct 30 00:28:52.947582 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.947359 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-93016b\"/\"praxis-route-93016b-kserve-self-signed-certs\"" Oct 30 00:28:52.947582 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.947476 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-93016b\"/\"openshift-service-ca.crt\"" Oct 30 00:28:52.949870 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.949033 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-93016b\"/\"kube-root-ca.crt\"" Oct 30 00:28:52.954166 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:52.954122 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0a4294e-62aa-49ea-928a-da94c5381802" path="/var/lib/kubelet/pods/c0a4294e-62aa-49ea-928a-da94c5381802/volumes" Oct 30 00:28:53.001596 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:53.001564 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec207720-74f5-4146-93cc-c633e090a5c8-tmp-dir\") pod \"praxis-route-93016b-kserve-f95f49c9-dn897\" (UID: \"ec207720-74f5-4146-93cc-c633e090a5c8\") " pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" Oct 30 00:28:53.001737 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:53.001615 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec207720-74f5-4146-93cc-c633e090a5c8-home\") pod \"praxis-route-93016b-kserve-f95f49c9-dn897\" (UID: \"ec207720-74f5-4146-93cc-c633e090a5c8\") " pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" Oct 30 00:28:53.001791 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:53.001736 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec207720-74f5-4146-93cc-c633e090a5c8-dshm\") pod \"praxis-route-93016b-kserve-f95f49c9-dn897\" (UID: \"ec207720-74f5-4146-93cc-c633e090a5c8\") " pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" Oct 30 00:28:53.001874 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:53.001857 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec207720-74f5-4146-93cc-c633e090a5c8-model-cache\") pod \"praxis-route-93016b-kserve-f95f49c9-dn897\" (UID: \"ec207720-74f5-4146-93cc-c633e090a5c8\") " pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" Oct 30 00:28:53.001967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:53.001948 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcnd6\" (UniqueName: \"kubernetes.io/projected/ec207720-74f5-4146-93cc-c633e090a5c8-kube-api-access-dcnd6\") pod \"praxis-route-93016b-kserve-f95f49c9-dn897\" (UID: \"ec207720-74f5-4146-93cc-c633e090a5c8\") " pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" Oct 30 00:28:53.002054 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:53.002042 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec207720-74f5-4146-93cc-c633e090a5c8-tls-certs\") pod \"praxis-route-93016b-kserve-f95f49c9-dn897\" (UID: \"ec207720-74f5-4146-93cc-c633e090a5c8\") " pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" Oct 30 00:28:53.002163 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:53.002151 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1359a6e8-77c9-4e05-b11e-27ba0c4573fe-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:53.002202 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:53.002166 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmz2p\" (UniqueName: \"kubernetes.io/projected/1359a6e8-77c9-4e05-b11e-27ba0c4573fe-kube-api-access-cmz2p\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:53.104353 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:53.104271 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec207720-74f5-4146-93cc-c633e090a5c8-dshm\") pod \"praxis-route-93016b-kserve-f95f49c9-dn897\" (UID: \"ec207720-74f5-4146-93cc-c633e090a5c8\") " pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" Oct 30 00:28:53.104526 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:53.104429 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec207720-74f5-4146-93cc-c633e090a5c8-model-cache\") pod \"praxis-route-93016b-kserve-f95f49c9-dn897\" (UID: \"ec207720-74f5-4146-93cc-c633e090a5c8\") " pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" Oct 30 00:28:53.104605 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:53.104545 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcnd6\" (UniqueName: \"kubernetes.io/projected/ec207720-74f5-4146-93cc-c633e090a5c8-kube-api-access-dcnd6\") pod \"praxis-route-93016b-kserve-f95f49c9-dn897\" (UID: \"ec207720-74f5-4146-93cc-c633e090a5c8\") " pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" Oct 30 00:28:53.104715 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:53.104650 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec207720-74f5-4146-93cc-c633e090a5c8-tls-certs\") pod \"praxis-route-93016b-kserve-f95f49c9-dn897\" (UID: \"ec207720-74f5-4146-93cc-c633e090a5c8\") " pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" Oct 30 00:28:53.104794 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:53.104780 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec207720-74f5-4146-93cc-c633e090a5c8-tmp-dir\") pod \"praxis-route-93016b-kserve-f95f49c9-dn897\" (UID: \"ec207720-74f5-4146-93cc-c633e090a5c8\") " pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" Oct 30 00:28:53.104878 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:53.104811 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec207720-74f5-4146-93cc-c633e090a5c8-model-cache\") pod \"praxis-route-93016b-kserve-f95f49c9-dn897\" (UID: \"ec207720-74f5-4146-93cc-c633e090a5c8\") " pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" Oct 30 00:28:53.104878 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:53.104825 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec207720-74f5-4146-93cc-c633e090a5c8-home\") pod \"praxis-route-93016b-kserve-f95f49c9-dn897\" (UID: \"ec207720-74f5-4146-93cc-c633e090a5c8\") " pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" Oct 30 00:28:53.106044 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:53.105205 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec207720-74f5-4146-93cc-c633e090a5c8-tmp-dir\") pod \"praxis-route-93016b-kserve-f95f49c9-dn897\" (UID: \"ec207720-74f5-4146-93cc-c633e090a5c8\") " pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" Oct 30 00:28:53.106044 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:53.105627 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec207720-74f5-4146-93cc-c633e090a5c8-home\") pod \"praxis-route-93016b-kserve-f95f49c9-dn897\" (UID: \"ec207720-74f5-4146-93cc-c633e090a5c8\") " pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" Oct 30 00:28:53.107039 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:53.107013 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec207720-74f5-4146-93cc-c633e090a5c8-dshm\") pod \"praxis-route-93016b-kserve-f95f49c9-dn897\" (UID: \"ec207720-74f5-4146-93cc-c633e090a5c8\") " pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" Oct 30 00:28:53.107857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:53.107815 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec207720-74f5-4146-93cc-c633e090a5c8-tls-certs\") pod \"praxis-route-93016b-kserve-f95f49c9-dn897\" (UID: \"ec207720-74f5-4146-93cc-c633e090a5c8\") " pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" Oct 30 00:28:53.180558 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:53.175934 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcnd6\" (UniqueName: \"kubernetes.io/projected/ec207720-74f5-4146-93cc-c633e090a5c8-kube-api-access-dcnd6\") pod \"praxis-route-93016b-kserve-f95f49c9-dn897\" (UID: \"ec207720-74f5-4146-93cc-c633e090a5c8\") " pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" Oct 30 00:28:53.265932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:53.265483 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" Oct 30 00:28:53.886378 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:53.886340 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-bf5958687-wqrzq" event={"ID":"1359a6e8-77c9-4e05-b11e-27ba0c4573fe","Type":"ContainerDied","Data":"6694143f2c848ba4e9f784f3f75a4adf6431d69e3ba5e8e69701f8a4aee8d3af"} Oct 30 00:28:53.886902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:53.886399 5001 scope.go:122] "RemoveContainer" containerID="347f96f80d12786d86391924e2b1e64b378ff8be8741652e052737c888f119a2" Oct 30 00:28:53.886902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:53.886530 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-bf5958687-wqrzq" Oct 30 00:28:53.935115 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:53.935079 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-bf5958687-wqrzq"] Oct 30 00:28:53.944852 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:53.944821 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f8397b-bf5958687-wqrzq"] Oct 30 00:28:53.954510 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:53.954472 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897"] Oct 30 00:28:54.899330 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:54.899293 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" event={"ID":"ec207720-74f5-4146-93cc-c633e090a5c8","Type":"ContainerStarted","Data":"490bd7fe56bdcdc42c260b44023919d340e9021fece84ceda155d1566d2885b5"} Oct 30 00:28:54.899330 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:54.899335 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" event={"ID":"ec207720-74f5-4146-93cc-c633e090a5c8","Type":"ContainerStarted","Data":"9f1436332a2930c3b0bbacf05acc59e3980be01e156d7ba550054091823035c0"} Oct 30 00:28:54.922756 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:54.922717 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1359a6e8-77c9-4e05-b11e-27ba0c4573fe" path="/var/lib/kubelet/pods/1359a6e8-77c9-4e05-b11e-27ba0c4573fe/volumes" Oct 30 00:28:54.939774 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:54.939724 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" podStartSLOduration=2.939710358 podStartE2EDuration="2.939710358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:28:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:28:54.935068279 +0000 UTC m=+1450.140569460" watchObservedRunningTime="2026-10-30 00:28:54.939710358 +0000 UTC m=+1450.145211566" Oct 30 00:28:55.266513 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:55.266476 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" Oct 30 00:28:56.894583 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:56.894559 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:28:56.923764 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:56.923735 5001 generic.go:356] "Generic (PLEG): container finished" podID="4fb15e40-3233-4329-917c-6bfcf33cb894" containerID="6d6b3c1b010268cd670524f9ab127b3501e621b4281a1f0a033aac64660208e5" exitCode=0 Oct 30 00:28:56.923929 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:56.923778 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" Oct 30 00:28:56.923929 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:56.923821 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" event={"ID":"4fb15e40-3233-4329-917c-6bfcf33cb894","Type":"ContainerDied","Data":"6d6b3c1b010268cd670524f9ab127b3501e621b4281a1f0a033aac64660208e5"} Oct 30 00:28:56.923929 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:56.923868 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8" event={"ID":"4fb15e40-3233-4329-917c-6bfcf33cb894","Type":"ContainerDied","Data":"a4b639d6487d66bb6bfbbf4c0c743bbababef912ed1a60e33d5d9d323d4b15d4"} Oct 30 00:28:56.923929 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:56.923885 5001 scope.go:122] "RemoveContainer" containerID="6d6b3c1b010268cd670524f9ab127b3501e621b4281a1f0a033aac64660208e5" Oct 30 00:28:56.925598 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:56.925570 5001 generic.go:356] "Generic (PLEG): container finished" podID="bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6" containerID="02b8bbb99fd50c202822d113a2aa0456014e60c70caaaa058a5d9e9a665e4e87" exitCode=0 Oct 30 00:28:56.925736 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:56.925652 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" event={"ID":"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6","Type":"ContainerDied","Data":"02b8bbb99fd50c202822d113a2aa0456014e60c70caaaa058a5d9e9a665e4e87"} Oct 30 00:28:56.926144 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:56.926126 5001 scope.go:122] "RemoveContainer" containerID="02b8bbb99fd50c202822d113a2aa0456014e60c70caaaa058a5d9e9a665e4e87" Oct 30 00:28:56.928157 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:56.928121 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:28:56.928247 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:56.928216 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:28:56.928298 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:56.928282 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:28:56.936270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:56.936251 5001 scope.go:122] "RemoveContainer" containerID="72ce3749930f3cca3cbe89433546c1d3a3949e2181da0067d680dad946f357b7" Oct 30 00:28:56.949249 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:56.949230 5001 scope.go:122] "RemoveContainer" containerID="6d6b3c1b010268cd670524f9ab127b3501e621b4281a1f0a033aac64660208e5" Oct 30 00:28:56.949672 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:28:56.949640 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d6b3c1b010268cd670524f9ab127b3501e621b4281a1f0a033aac64660208e5\": container with ID starting with 6d6b3c1b010268cd670524f9ab127b3501e621b4281a1f0a033aac64660208e5 not found: ID does not exist" containerID="6d6b3c1b010268cd670524f9ab127b3501e621b4281a1f0a033aac64660208e5" Oct 30 00:28:56.949778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:56.949682 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d6b3c1b010268cd670524f9ab127b3501e621b4281a1f0a033aac64660208e5"} err="failed to get container status \"6d6b3c1b010268cd670524f9ab127b3501e621b4281a1f0a033aac64660208e5\": rpc error: code = NotFound desc = could not find container \"6d6b3c1b010268cd670524f9ab127b3501e621b4281a1f0a033aac64660208e5\": container with ID starting with 6d6b3c1b010268cd670524f9ab127b3501e621b4281a1f0a033aac64660208e5 not found: ID does not exist" Oct 30 00:28:56.949778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:56.949705 5001 scope.go:122] "RemoveContainer" containerID="72ce3749930f3cca3cbe89433546c1d3a3949e2181da0067d680dad946f357b7" Oct 30 00:28:56.950756 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:28:56.950730 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72ce3749930f3cca3cbe89433546c1d3a3949e2181da0067d680dad946f357b7\": container with ID starting with 72ce3749930f3cca3cbe89433546c1d3a3949e2181da0067d680dad946f357b7 not found: ID does not exist" containerID="72ce3749930f3cca3cbe89433546c1d3a3949e2181da0067d680dad946f357b7" Oct 30 00:28:56.950884 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:56.950768 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72ce3749930f3cca3cbe89433546c1d3a3949e2181da0067d680dad946f357b7"} err="failed to get container status \"72ce3749930f3cca3cbe89433546c1d3a3949e2181da0067d680dad946f357b7\": rpc error: code = NotFound desc = could not find container \"72ce3749930f3cca3cbe89433546c1d3a3949e2181da0067d680dad946f357b7\": container with ID starting with 72ce3749930f3cca3cbe89433546c1d3a3949e2181da0067d680dad946f357b7 not found: ID does not exist" Oct 30 00:28:57.013711 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.013685 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-workload-certs\") pod \"4fb15e40-3233-4329-917c-6bfcf33cb894\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " Oct 30 00:28:57.013894 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.013736 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fb15e40-3233-4329-917c-6bfcf33cb894-istio-token\" (UniqueName: \"kubernetes.io/projected/4fb15e40-3233-4329-917c-6bfcf33cb894-istio-token\") pod \"4fb15e40-3233-4329-917c-6bfcf33cb894\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " Oct 30 00:28:57.013894 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.013834 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-istio-envoy\") pod \"4fb15e40-3233-4329-917c-6bfcf33cb894\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " Oct 30 00:28:57.014320 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.014290 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-workload-certs" pod "4fb15e40-3233-4329-917c-6bfcf33cb894" (UID: "4fb15e40-3233-4329-917c-6bfcf33cb894"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:28:57.014427 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.014363 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fb15e40-3233-4329-917c-6bfcf33cb894-istiod-ca-cert" pod "4fb15e40-3233-4329-917c-6bfcf33cb894" (UID: "4fb15e40-3233-4329-917c-6bfcf33cb894"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:28:57.014427 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.014398 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fb15e40-3233-4329-917c-6bfcf33cb894-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fb15e40-3233-4329-917c-6bfcf33cb894-istiod-ca-cert\") pod \"4fb15e40-3233-4329-917c-6bfcf33cb894\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " Oct 30 00:28:57.014562 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.014470 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-istio-data\") pod \"4fb15e40-3233-4329-917c-6bfcf33cb894\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " Oct 30 00:28:57.014621 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.014578 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4fb15e40-3233-4329-917c-6bfcf33cb894-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fb15e40-3233-4329-917c-6bfcf33cb894-istio-podinfo\") pod \"4fb15e40-3233-4329-917c-6bfcf33cb894\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " Oct 30 00:28:57.014756 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.014741 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-credential-socket\") pod \"4fb15e40-3233-4329-917c-6bfcf33cb894\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " Oct 30 00:28:57.014813 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.014795 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-workload-socket\") pod \"4fb15e40-3233-4329-917c-6bfcf33cb894\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " Oct 30 00:28:57.015356 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.014815 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fb15e40-3233-4329-917c-6bfcf33cb894-kube-api-access-nks6s\" (UniqueName: \"kubernetes.io/projected/4fb15e40-3233-4329-917c-6bfcf33cb894-kube-api-access-nks6s\") pod \"4fb15e40-3233-4329-917c-6bfcf33cb894\" (UID: \"4fb15e40-3233-4329-917c-6bfcf33cb894\") " Oct 30 00:28:57.015356 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.014738 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-istio-data" pod "4fb15e40-3233-4329-917c-6bfcf33cb894" (UID: "4fb15e40-3233-4329-917c-6bfcf33cb894"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:28:57.016834 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.016074 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:57.016834 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.016101 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fb15e40-3233-4329-917c-6bfcf33cb894-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:57.016834 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.016116 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:57.017120 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.017080 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-workload-socket" pod "4fb15e40-3233-4329-917c-6bfcf33cb894" (UID: "4fb15e40-3233-4329-917c-6bfcf33cb894"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:28:57.017343 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.017318 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-credential-socket" pod "4fb15e40-3233-4329-917c-6bfcf33cb894" (UID: "4fb15e40-3233-4329-917c-6bfcf33cb894"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:28:57.017754 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.017705 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fb15e40-3233-4329-917c-6bfcf33cb894-istio-token" pod "4fb15e40-3233-4329-917c-6bfcf33cb894" (UID: "4fb15e40-3233-4329-917c-6bfcf33cb894"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:28:57.019406 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.018425 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4fb15e40-3233-4329-917c-6bfcf33cb894-istio-podinfo" pod "4fb15e40-3233-4329-917c-6bfcf33cb894" (UID: "4fb15e40-3233-4329-917c-6bfcf33cb894"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:28:57.021048 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.020422 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fb15e40-3233-4329-917c-6bfcf33cb894-kube-api-access-nks6s" pod "4fb15e40-3233-4329-917c-6bfcf33cb894" (UID: "4fb15e40-3233-4329-917c-6bfcf33cb894"). InnerVolumeSpecName "kube-api-access-nks6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:28:57.021048 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.021013 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-istio-envoy" pod "4fb15e40-3233-4329-917c-6bfcf33cb894" (UID: "4fb15e40-3233-4329-917c-6bfcf33cb894"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:28:57.118429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.118389 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:57.118429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.118429 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fb15e40-3233-4329-917c-6bfcf33cb894-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:57.118697 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.118443 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:57.118697 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.118458 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb15e40-3233-4329-917c-6bfcf33cb894-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:57.118697 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.118472 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nks6s\" (UniqueName: \"kubernetes.io/projected/4fb15e40-3233-4329-917c-6bfcf33cb894-kube-api-access-nks6s\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:57.118697 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.118484 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fb15e40-3233-4329-917c-6bfcf33cb894-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:28:57.270351 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.270318 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8"] Oct 30 00:28:57.274933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.274889 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-522ea7-openshift-default-75dfc8b4f-xxfk8"] Oct 30 00:28:57.416357 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.416324 5001 scope.go:122] "RemoveContainer" containerID="0f47b897bf162924d8ad4be0a94889db4492558fba8db636ede7bcfdf63b8f59" Oct 30 00:28:57.448086 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.448045 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4500d5-x6w85"] Oct 30 00:28:57.448864 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.448844 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4fb15e40-3233-4329-917c-6bfcf33cb894" containerName="istio-proxy" Oct 30 00:28:57.449171 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.448867 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fb15e40-3233-4329-917c-6bfcf33cb894" containerName="istio-proxy" Oct 30 00:28:57.449171 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.449088 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fb15e40-3233-4329-917c-6bfcf33cb894" containerName="istio-proxy" Oct 30 00:28:57.449171 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.449103 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fb15e40-3233-4329-917c-6bfcf33cb894" containerName="istio-proxy" Oct 30 00:28:57.452391 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.452372 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4500d5-x6w85" Oct 30 00:28:57.467427 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.466397 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:28:57.467427 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.466434 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:28:57.467427 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.466469 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:28:57.467427 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.466501 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:28:57.472848 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.472827 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4500d5-x6w85"] Oct 30 00:28:57.531074 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.531039 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf6bp\" (UniqueName: \"kubernetes.io/projected/21ee8ab8-ee55-4801-a39e-8d5ad1f596ee-kube-api-access-pf6bp\") pod \"maas-api-revoke-keys-e2e-praxis-b-4500d5-x6w85\" (UID: \"21ee8ab8-ee55-4801-a39e-8d5ad1f596ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4500d5-x6w85" Oct 30 00:28:57.531333 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.531306 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ee8ab8-ee55-4801-a39e-8d5ad1f596ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4500d5-x6w85\" (UID: \"21ee8ab8-ee55-4801-a39e-8d5ad1f596ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4500d5-x6w85" Oct 30 00:28:57.531445 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.531347 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ee8ab8-ee55-4801-a39e-8d5ad1f596ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4500d5-x6w85\" (UID: \"21ee8ab8-ee55-4801-a39e-8d5ad1f596ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4500d5-x6w85" Oct 30 00:28:57.633022 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.632989 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf6bp\" (UniqueName: \"kubernetes.io/projected/21ee8ab8-ee55-4801-a39e-8d5ad1f596ee-kube-api-access-pf6bp\") pod \"maas-api-revoke-keys-e2e-praxis-b-4500d5-x6w85\" (UID: \"21ee8ab8-ee55-4801-a39e-8d5ad1f596ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4500d5-x6w85" Oct 30 00:28:57.633171 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.633109 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ee8ab8-ee55-4801-a39e-8d5ad1f596ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4500d5-x6w85\" (UID: \"21ee8ab8-ee55-4801-a39e-8d5ad1f596ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4500d5-x6w85" Oct 30 00:28:57.633171 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.633140 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ee8ab8-ee55-4801-a39e-8d5ad1f596ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4500d5-x6w85\" (UID: \"21ee8ab8-ee55-4801-a39e-8d5ad1f596ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4500d5-x6w85" Oct 30 00:28:57.633934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.633896 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ee8ab8-ee55-4801-a39e-8d5ad1f596ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4500d5-x6w85\" (UID: \"21ee8ab8-ee55-4801-a39e-8d5ad1f596ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4500d5-x6w85" Oct 30 00:28:57.635642 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.635614 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ee8ab8-ee55-4801-a39e-8d5ad1f596ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4500d5-x6w85\" (UID: \"21ee8ab8-ee55-4801-a39e-8d5ad1f596ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4500d5-x6w85" Oct 30 00:28:57.655427 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.655399 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf6bp\" (UniqueName: \"kubernetes.io/projected/21ee8ab8-ee55-4801-a39e-8d5ad1f596ee-kube-api-access-pf6bp\") pod \"maas-api-revoke-keys-e2e-praxis-b-4500d5-x6w85\" (UID: \"21ee8ab8-ee55-4801-a39e-8d5ad1f596ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4500d5-x6w85" Oct 30 00:28:57.772104 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.772076 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4500d5-x6w85" Oct 30 00:28:57.944183 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:57.944141 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" event={"ID":"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6","Type":"ContainerStarted","Data":"6e24e1e71d99bfc753ed6edeee268e066803b81ec9f2b578341e7f9966b98114"} Oct 30 00:28:58.090174 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:28:58.090147 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ee8ab8_ee55_4801_a39e_8d5ad1f596ee.slice/crio-90f71f6069b876340ac299619b82965cf08550eb16194419eff29752758f47c4 WatchSource:0}: Error finding container 90f71f6069b876340ac299619b82965cf08550eb16194419eff29752758f47c4: Status 404 returned error can't find the container with id 90f71f6069b876340ac299619b82965cf08550eb16194419eff29752758f47c4 Oct 30 00:28:58.090527 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:58.090504 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4500d5-x6w85"] Oct 30 00:28:58.889212 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:58.889176 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:28:58.890892 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:58.890858 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 00:28:58.891074 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:58.890950 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" podUID="bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 00:28:59.031146 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:59.031106 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fb15e40-3233-4329-917c-6bfcf33cb894" path="/var/lib/kubelet/pods/4fb15e40-3233-4329-917c-6bfcf33cb894/volumes" Oct 30 00:28:59.087938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:59.084193 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4500d5-x6w85" event={"ID":"21ee8ab8-ee55-4801-a39e-8d5ad1f596ee","Type":"ContainerStarted","Data":"afc79742de4339f9215ff9872320437f83deb06c02c494946c4aec766f5338f5"} Oct 30 00:28:59.087938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:59.084252 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4500d5-x6w85" event={"ID":"21ee8ab8-ee55-4801-a39e-8d5ad1f596ee","Type":"ContainerStarted","Data":"90f71f6069b876340ac299619b82965cf08550eb16194419eff29752758f47c4"} Oct 30 00:28:59.154505 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:59.154396 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4500d5-x6w85" podStartSLOduration=2.154379953 podStartE2EDuration="2.154379953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:28:59.14233088 +0000 UTC m=+1454.347832063" watchObservedRunningTime="2026-10-30 00:28:59.154379953 +0000 UTC m=+1454.359881166" Oct 30 00:28:59.253940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:59.243996 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l"] Oct 30 00:28:59.257305 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:59.257213 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" containerID="cri-o://0dfbf990a78ac40811b9bb64141e12639acd028f98846a2c82fd321d74994aa1" gracePeriod=30 Oct 30 00:28:59.785880 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:59.784773 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a15230-kt696"] Oct 30 00:28:59.785880 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:59.785528 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4fb15e40-3233-4329-917c-6bfcf33cb894" containerName="istio-proxy" Oct 30 00:28:59.785880 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:59.785543 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fb15e40-3233-4329-917c-6bfcf33cb894" containerName="istio-proxy" Oct 30 00:28:59.788406 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:59.788387 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a15230-kt696" Oct 30 00:28:59.797770 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:59.797741 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a15230-kt696"] Oct 30 00:28:59.889521 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:59.889486 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 00:28:59.889707 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:59.889537 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" podUID="bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 00:28:59.892393 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:59.892365 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljc7z\" (UniqueName: \"kubernetes.io/projected/3ae0b29f-6fd9-494f-b22c-e313c324615f-kube-api-access-ljc7z\") pod \"maas-api-revoke-keys-e2e-ipp-b-a15230-kt696\" (UID: \"3ae0b29f-6fd9-494f-b22c-e313c324615f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a15230-kt696" Oct 30 00:28:59.892531 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:59.892424 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ae0b29f-6fd9-494f-b22c-e313c324615f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a15230-kt696\" (UID: \"3ae0b29f-6fd9-494f-b22c-e313c324615f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a15230-kt696" Oct 30 00:28:59.892591 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:59.892555 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ae0b29f-6fd9-494f-b22c-e313c324615f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a15230-kt696\" (UID: \"3ae0b29f-6fd9-494f-b22c-e313c324615f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a15230-kt696" Oct 30 00:28:59.994556 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:59.994522 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ae0b29f-6fd9-494f-b22c-e313c324615f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a15230-kt696\" (UID: \"3ae0b29f-6fd9-494f-b22c-e313c324615f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a15230-kt696" Oct 30 00:28:59.994750 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:59.994734 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljc7z\" (UniqueName: \"kubernetes.io/projected/3ae0b29f-6fd9-494f-b22c-e313c324615f-kube-api-access-ljc7z\") pod \"maas-api-revoke-keys-e2e-ipp-b-a15230-kt696\" (UID: \"3ae0b29f-6fd9-494f-b22c-e313c324615f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a15230-kt696" Oct 30 00:28:59.994800 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:59.994781 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ae0b29f-6fd9-494f-b22c-e313c324615f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a15230-kt696\" (UID: \"3ae0b29f-6fd9-494f-b22c-e313c324615f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a15230-kt696" Oct 30 00:28:59.995187 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:59.995166 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ae0b29f-6fd9-494f-b22c-e313c324615f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a15230-kt696\" (UID: \"3ae0b29f-6fd9-494f-b22c-e313c324615f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a15230-kt696" Oct 30 00:28:59.997041 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:28:59.997018 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ae0b29f-6fd9-494f-b22c-e313c324615f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a15230-kt696\" (UID: \"3ae0b29f-6fd9-494f-b22c-e313c324615f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a15230-kt696" Oct 30 00:29:00.037956 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:00.037861 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljc7z\" (UniqueName: \"kubernetes.io/projected/3ae0b29f-6fd9-494f-b22c-e313c324615f-kube-api-access-ljc7z\") pod \"maas-api-revoke-keys-e2e-ipp-b-a15230-kt696\" (UID: \"3ae0b29f-6fd9-494f-b22c-e313c324615f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a15230-kt696" Oct 30 00:29:00.093417 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:00.093384 5001 generic.go:356] "Generic (PLEG): container finished" podID="21ee8ab8-ee55-4801-a39e-8d5ad1f596ee" containerID="afc79742de4339f9215ff9872320437f83deb06c02c494946c4aec766f5338f5" exitCode=0 Oct 30 00:29:00.093610 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:00.093462 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4500d5-x6w85" event={"ID":"21ee8ab8-ee55-4801-a39e-8d5ad1f596ee","Type":"ContainerDied","Data":"afc79742de4339f9215ff9872320437f83deb06c02c494946c4aec766f5338f5"} Oct 30 00:29:00.107628 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:00.107601 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a15230-kt696" Oct 30 00:29:00.460249 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:29:00.460222 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae0b29f_6fd9_494f_b22c_e313c324615f.slice/crio-75efa1045cf94e38e65804592c5c3f0c5d4bacf67c7bcdd0630f94ea3713bb65 WatchSource:0}: Error finding container 75efa1045cf94e38e65804592c5c3f0c5d4bacf67c7bcdd0630f94ea3713bb65: Status 404 returned error can't find the container with id 75efa1045cf94e38e65804592c5c3f0c5d4bacf67c7bcdd0630f94ea3713bb65 Oct 30 00:29:00.460428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:00.460404 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a15230-kt696"] Oct 30 00:29:00.890219 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:00.890187 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 00:29:00.890367 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:00.890234 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" podUID="bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 00:29:01.106046 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:01.105953 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a15230-kt696" event={"ID":"3ae0b29f-6fd9-494f-b22c-e313c324615f","Type":"ContainerStarted","Data":"9c9744aba98cca178108371d9aa83cc4c1f16c71db773c7e239c9254e952ed35"} Oct 30 00:29:01.106046 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:01.106023 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a15230-kt696" event={"ID":"3ae0b29f-6fd9-494f-b22c-e313c324615f","Type":"ContainerStarted","Data":"75efa1045cf94e38e65804592c5c3f0c5d4bacf67c7bcdd0630f94ea3713bb65"} Oct 30 00:29:01.152116 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:01.152062 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a15230-kt696" podStartSLOduration=2.152047308 podStartE2EDuration="2.152047308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:29:01.147648769 +0000 UTC m=+1456.353149945" watchObservedRunningTime="2026-10-30 00:29:01.152047308 +0000 UTC m=+1456.357548489" Oct 30 00:29:01.343752 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:01.343727 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4500d5-x6w85" Oct 30 00:29:01.494108 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:01.494031 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21ee8ab8-ee55-4801-a39e-8d5ad1f596ee-kube-api-access-pf6bp\" (UniqueName: \"kubernetes.io/projected/21ee8ab8-ee55-4801-a39e-8d5ad1f596ee-kube-api-access-pf6bp\") pod \"21ee8ab8-ee55-4801-a39e-8d5ad1f596ee\" (UID: \"21ee8ab8-ee55-4801-a39e-8d5ad1f596ee\") " Oct 30 00:29:01.494263 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:01.494147 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21ee8ab8-ee55-4801-a39e-8d5ad1f596ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ee8ab8-ee55-4801-a39e-8d5ad1f596ee-maas-api-openshift-ca\") pod \"21ee8ab8-ee55-4801-a39e-8d5ad1f596ee\" (UID: \"21ee8ab8-ee55-4801-a39e-8d5ad1f596ee\") " Oct 30 00:29:01.494263 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:01.494192 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21ee8ab8-ee55-4801-a39e-8d5ad1f596ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ee8ab8-ee55-4801-a39e-8d5ad1f596ee-maas-api-xks-ca\") pod \"21ee8ab8-ee55-4801-a39e-8d5ad1f596ee\" (UID: \"21ee8ab8-ee55-4801-a39e-8d5ad1f596ee\") " Oct 30 00:29:01.494508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:01.494476 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21ee8ab8-ee55-4801-a39e-8d5ad1f596ee-maas-api-openshift-ca" pod "21ee8ab8-ee55-4801-a39e-8d5ad1f596ee" (UID: "21ee8ab8-ee55-4801-a39e-8d5ad1f596ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:29:01.494820 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:01.494800 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ee8ab8-ee55-4801-a39e-8d5ad1f596ee-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:01.496217 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:01.496197 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ee8ab8-ee55-4801-a39e-8d5ad1f596ee-kube-api-access-pf6bp" pod "21ee8ab8-ee55-4801-a39e-8d5ad1f596ee" (UID: "21ee8ab8-ee55-4801-a39e-8d5ad1f596ee"). InnerVolumeSpecName "kube-api-access-pf6bp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:29:01.496313 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:01.496288 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21ee8ab8-ee55-4801-a39e-8d5ad1f596ee-maas-api-xks-ca" pod "21ee8ab8-ee55-4801-a39e-8d5ad1f596ee" (UID: "21ee8ab8-ee55-4801-a39e-8d5ad1f596ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:29:01.596013 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:01.595980 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf6bp\" (UniqueName: \"kubernetes.io/projected/21ee8ab8-ee55-4801-a39e-8d5ad1f596ee-kube-api-access-pf6bp\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:01.596013 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:01.596011 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ee8ab8-ee55-4801-a39e-8d5ad1f596ee-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:01.883561 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:01.883529 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:29:01.889843 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:01.889817 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 00:29:01.889975 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:01.889874 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" podUID="bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 00:29:02.151668 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:02.151581 5001 generic.go:356] "Generic (PLEG): container finished" podID="3ae0b29f-6fd9-494f-b22c-e313c324615f" containerID="9c9744aba98cca178108371d9aa83cc4c1f16c71db773c7e239c9254e952ed35" exitCode=0 Oct 30 00:29:02.152176 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:02.151654 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a15230-kt696" event={"ID":"3ae0b29f-6fd9-494f-b22c-e313c324615f","Type":"ContainerDied","Data":"9c9744aba98cca178108371d9aa83cc4c1f16c71db773c7e239c9254e952ed35"} Oct 30 00:29:02.153708 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:02.153687 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4500d5-x6w85" Oct 30 00:29:02.153849 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:02.153688 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4500d5-x6w85" event={"ID":"21ee8ab8-ee55-4801-a39e-8d5ad1f596ee","Type":"ContainerDied","Data":"90f71f6069b876340ac299619b82965cf08550eb16194419eff29752758f47c4"} Oct 30 00:29:02.153849 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:02.153803 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90f71f6069b876340ac299619b82965cf08550eb16194419eff29752758f47c4" Oct 30 00:29:02.892943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:02.890365 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 00:29:02.892943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:02.890436 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" podUID="bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 00:29:03.266482 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:03.266444 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" Oct 30 00:29:03.283414 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:03.282945 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" Oct 30 00:29:03.389129 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:03.389097 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a15230-kt696" Oct 30 00:29:03.545735 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:03.545701 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ae0b29f-6fd9-494f-b22c-e313c324615f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ae0b29f-6fd9-494f-b22c-e313c324615f-maas-api-xks-ca\") pod \"3ae0b29f-6fd9-494f-b22c-e313c324615f\" (UID: \"3ae0b29f-6fd9-494f-b22c-e313c324615f\") " Oct 30 00:29:03.545940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:03.545793 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ae0b29f-6fd9-494f-b22c-e313c324615f-kube-api-access-ljc7z\" (UniqueName: \"kubernetes.io/projected/3ae0b29f-6fd9-494f-b22c-e313c324615f-kube-api-access-ljc7z\") pod \"3ae0b29f-6fd9-494f-b22c-e313c324615f\" (UID: \"3ae0b29f-6fd9-494f-b22c-e313c324615f\") " Oct 30 00:29:03.545940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:03.545869 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ae0b29f-6fd9-494f-b22c-e313c324615f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ae0b29f-6fd9-494f-b22c-e313c324615f-maas-api-openshift-ca\") pod \"3ae0b29f-6fd9-494f-b22c-e313c324615f\" (UID: \"3ae0b29f-6fd9-494f-b22c-e313c324615f\") " Oct 30 00:29:03.546277 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:03.546252 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ae0b29f-6fd9-494f-b22c-e313c324615f-maas-api-openshift-ca" pod "3ae0b29f-6fd9-494f-b22c-e313c324615f" (UID: "3ae0b29f-6fd9-494f-b22c-e313c324615f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:29:03.546992 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:03.546966 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ae0b29f-6fd9-494f-b22c-e313c324615f-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:03.547973 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:03.547952 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ae0b29f-6fd9-494f-b22c-e313c324615f-maas-api-xks-ca" pod "3ae0b29f-6fd9-494f-b22c-e313c324615f" (UID: "3ae0b29f-6fd9-494f-b22c-e313c324615f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:29:03.548244 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:03.548223 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ae0b29f-6fd9-494f-b22c-e313c324615f-kube-api-access-ljc7z" pod "3ae0b29f-6fd9-494f-b22c-e313c324615f" (UID: "3ae0b29f-6fd9-494f-b22c-e313c324615f"). InnerVolumeSpecName "kube-api-access-ljc7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:29:03.648587 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:03.648550 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ae0b29f-6fd9-494f-b22c-e313c324615f-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:03.648587 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:03.648577 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljc7z\" (UniqueName: \"kubernetes.io/projected/3ae0b29f-6fd9-494f-b22c-e313c324615f-kube-api-access-ljc7z\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:03.750787 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:03.750752 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:29:03.751014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:03.750816 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:29:03.889842 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:03.889749 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 00:29:03.889842 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:03.889817 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" podUID="bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 00:29:04.374765 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:04.315769 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a15230-kt696" Oct 30 00:29:04.374765 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:04.344108 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a15230-kt696" event={"ID":"3ae0b29f-6fd9-494f-b22c-e313c324615f","Type":"ContainerDied","Data":"75efa1045cf94e38e65804592c5c3f0c5d4bacf67c7bcdd0630f94ea3713bb65"} Oct 30 00:29:04.374765 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:04.344182 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75efa1045cf94e38e65804592c5c3f0c5d4bacf67c7bcdd0630f94ea3713bb65" Oct 30 00:29:04.482011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:04.474974 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cfcf7-bm58n"] Oct 30 00:29:04.482011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:04.477112 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cfcf7-bm58n"] Oct 30 00:29:04.557934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:04.557191 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" Oct 30 00:29:04.901794 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:04.897257 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:29:04.901794 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:04.900084 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:29:04.926717 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:04.926689 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a243f64a-3cd5-46b4-8f02-27b25c1d6e87" path="/var/lib/kubelet/pods/a243f64a-3cd5-46b4-8f02-27b25c1d6e87/volumes" Oct 30 00:29:05.048997 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.048972 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:29:05.144611 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.144576 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istio-token\" (UniqueName: \"kubernetes.io/projected/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istio-token\") pod \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " Oct 30 00:29:05.144781 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.144613 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istio-podinfo\") pod \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " Oct 30 00:29:05.144781 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.144693 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istio-envoy\") pod \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " Oct 30 00:29:05.144901 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.144787 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istiod-ca-cert\") pod \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " Oct 30 00:29:05.144901 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.144835 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-workload-certs\") pod \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " Oct 30 00:29:05.145056 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.144968 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-credential-socket\") pod \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " Oct 30 00:29:05.145056 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.145038 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-workload-socket\") pod \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " Oct 30 00:29:05.145169 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.145133 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istio-data\") pod \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " Oct 30 00:29:05.145225 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.145192 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-workload-certs" pod "9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" (UID: "9a6c9a83-505d-4955-b93f-ec44cf7b5cdb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:29:05.145225 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.145217 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-kube-api-access-xgg26\" (UniqueName: \"kubernetes.io/projected/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-kube-api-access-xgg26\") pod \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\" (UID: \"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb\") " Oct 30 00:29:05.145352 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.145323 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-credential-socket" pod "9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" (UID: "9a6c9a83-505d-4955-b93f-ec44cf7b5cdb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:29:05.145427 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.145263 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istiod-ca-cert" pod "9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" (UID: "9a6c9a83-505d-4955-b93f-ec44cf7b5cdb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:29:05.145500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.145473 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-workload-socket" pod "9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" (UID: "9a6c9a83-505d-4955-b93f-ec44cf7b5cdb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:29:05.145598 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.145572 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istio-data" pod "9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" (UID: "9a6c9a83-505d-4955-b93f-ec44cf7b5cdb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:29:05.146249 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.146179 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:05.146249 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.146216 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:05.146249 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.146231 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:05.146249 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.146246 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:05.146559 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.146262 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:05.147256 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.147230 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istio-podinfo" pod "9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" (UID: "9a6c9a83-505d-4955-b93f-ec44cf7b5cdb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:29:05.147711 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.147687 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-kube-api-access-xgg26" pod "9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" (UID: "9a6c9a83-505d-4955-b93f-ec44cf7b5cdb"). InnerVolumeSpecName "kube-api-access-xgg26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:29:05.147877 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.147859 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istio-token" pod "9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" (UID: "9a6c9a83-505d-4955-b93f-ec44cf7b5cdb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:29:05.147966 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.147945 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istio-envoy" pod "9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" (UID: "9a6c9a83-505d-4955-b93f-ec44cf7b5cdb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:29:05.248269 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.248236 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgg26\" (UniqueName: \"kubernetes.io/projected/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-kube-api-access-xgg26\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:05.248269 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.248266 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:05.248269 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.248277 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:05.248506 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.248286 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:05.325927 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.325883 5001 generic.go:356] "Generic (PLEG): container finished" podID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerID="0dfbf990a78ac40811b9bb64141e12639acd028f98846a2c82fd321d74994aa1" exitCode=0 Oct 30 00:29:05.326114 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.325949 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" Oct 30 00:29:05.326114 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.325942 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" event={"ID":"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb","Type":"ContainerDied","Data":"0dfbf990a78ac40811b9bb64141e12639acd028f98846a2c82fd321d74994aa1"} Oct 30 00:29:05.326114 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.326106 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l" event={"ID":"9a6c9a83-505d-4955-b93f-ec44cf7b5cdb","Type":"ContainerDied","Data":"1f85e49783fe866d42ff8785dae084b430799d61291cc4bd688ab5b8701462f4"} Oct 30 00:29:05.326305 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.326135 5001 scope.go:122] "RemoveContainer" containerID="0dfbf990a78ac40811b9bb64141e12639acd028f98846a2c82fd321d74994aa1" Oct 30 00:29:05.338741 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.338705 5001 scope.go:122] "RemoveContainer" containerID="7735f81688c824ce4c03de2ab668e577ef7e34cae26d89e81364b4b573038fcc" Oct 30 00:29:05.349507 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.349488 5001 scope.go:122] "RemoveContainer" containerID="0dfbf990a78ac40811b9bb64141e12639acd028f98846a2c82fd321d74994aa1" Oct 30 00:29:05.349756 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:29:05.349732 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dfbf990a78ac40811b9bb64141e12639acd028f98846a2c82fd321d74994aa1\": container with ID starting with 0dfbf990a78ac40811b9bb64141e12639acd028f98846a2c82fd321d74994aa1 not found: ID does not exist" containerID="0dfbf990a78ac40811b9bb64141e12639acd028f98846a2c82fd321d74994aa1" Oct 30 00:29:05.349806 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.349765 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dfbf990a78ac40811b9bb64141e12639acd028f98846a2c82fd321d74994aa1"} err="failed to get container status \"0dfbf990a78ac40811b9bb64141e12639acd028f98846a2c82fd321d74994aa1\": rpc error: code = NotFound desc = could not find container \"0dfbf990a78ac40811b9bb64141e12639acd028f98846a2c82fd321d74994aa1\": container with ID starting with 0dfbf990a78ac40811b9bb64141e12639acd028f98846a2c82fd321d74994aa1 not found: ID does not exist" Oct 30 00:29:05.349806 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.349782 5001 scope.go:122] "RemoveContainer" containerID="7735f81688c824ce4c03de2ab668e577ef7e34cae26d89e81364b4b573038fcc" Oct 30 00:29:05.350039 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:29:05.350022 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7735f81688c824ce4c03de2ab668e577ef7e34cae26d89e81364b4b573038fcc\": container with ID starting with 7735f81688c824ce4c03de2ab668e577ef7e34cae26d89e81364b4b573038fcc not found: ID does not exist" containerID="7735f81688c824ce4c03de2ab668e577ef7e34cae26d89e81364b4b573038fcc" Oct 30 00:29:05.350097 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.350045 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7735f81688c824ce4c03de2ab668e577ef7e34cae26d89e81364b4b573038fcc"} err="failed to get container status \"7735f81688c824ce4c03de2ab668e577ef7e34cae26d89e81364b4b573038fcc\": rpc error: code = NotFound desc = could not find container \"7735f81688c824ce4c03de2ab668e577ef7e34cae26d89e81364b4b573038fcc\": container with ID starting with 7735f81688c824ce4c03de2ab668e577ef7e34cae26d89e81364b4b573038fcc not found: ID does not exist" Oct 30 00:29:05.373283 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.373248 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l"] Oct 30 00:29:05.377701 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:05.377681 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f8397b-openshift-default-6d9c597fcc-jch7l"] Oct 30 00:29:06.217048 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.214964 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm"] Oct 30 00:29:06.217048 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.216049 5001 cpu_manager.go:414] "containerMap: removing container" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" Oct 30 00:29:06.217048 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.216069 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" Oct 30 00:29:06.217048 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.216092 5001 cpu_manager.go:414] "containerMap: removing container" podUID="3ae0b29f-6fd9-494f-b22c-e313c324615f" containerName="revoke-keys" Oct 30 00:29:06.217048 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.216099 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ae0b29f-6fd9-494f-b22c-e313c324615f" containerName="revoke-keys" Oct 30 00:29:06.217048 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.216138 5001 cpu_manager.go:414] "containerMap: removing container" podUID="21ee8ab8-ee55-4801-a39e-8d5ad1f596ee" containerName="revoke-keys" Oct 30 00:29:06.217048 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.216145 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21ee8ab8-ee55-4801-a39e-8d5ad1f596ee" containerName="revoke-keys" Oct 30 00:29:06.217048 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.216165 5001 cpu_manager.go:414] "containerMap: removing container" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" Oct 30 00:29:06.217048 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.216173 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" Oct 30 00:29:06.217048 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.216351 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" Oct 30 00:29:06.217048 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.216372 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" containerName="istio-proxy" Oct 30 00:29:06.217048 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.216383 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ae0b29f-6fd9-494f-b22c-e313c324615f" containerName="revoke-keys" Oct 30 00:29:06.217048 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.216397 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="21ee8ab8-ee55-4801-a39e-8d5ad1f596ee" containerName="revoke-keys" Oct 30 00:29:06.250356 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.230137 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm"] Oct 30 00:29:06.250356 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.230260 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:29:06.251616 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.251519 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3fb3ad-openshift-default-dockercfg-mwsjt\"" Oct 30 00:29:06.307029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.306986 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-istio-data\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:29:06.307411 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.307378 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-workload-socket\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:29:06.307569 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.307544 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44db3a17-452a-4667-bbc8-0f73b1711233-istio-token\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:29:06.307812 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.307795 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44db3a17-452a-4667-bbc8-0f73b1711233-istiod-ca-cert\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:29:06.307958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.307941 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-credential-socket\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:29:06.308162 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.308142 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-workload-certs\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:29:06.308327 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.308310 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-istio-envoy\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:29:06.308554 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.308533 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt9kx\" (UniqueName: \"kubernetes.io/projected/44db3a17-452a-4667-bbc8-0f73b1711233-kube-api-access-jt9kx\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:29:06.314012 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.313976 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44db3a17-452a-4667-bbc8-0f73b1711233-istio-podinfo\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:29:06.417937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.416877 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-istio-envoy\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:29:06.417937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.417057 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt9kx\" (UniqueName: \"kubernetes.io/projected/44db3a17-452a-4667-bbc8-0f73b1711233-kube-api-access-jt9kx\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:29:06.417937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.417088 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44db3a17-452a-4667-bbc8-0f73b1711233-istio-podinfo\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:29:06.417937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.417148 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-istio-data\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:29:06.417937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.417261 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-workload-socket\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:29:06.417937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.417286 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44db3a17-452a-4667-bbc8-0f73b1711233-istio-token\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:29:06.417937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.417374 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44db3a17-452a-4667-bbc8-0f73b1711233-istiod-ca-cert\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:29:06.417937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.417410 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-credential-socket\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:29:06.417937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.417515 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-workload-certs\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:29:06.418946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.418005 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-workload-certs\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:29:06.420224 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.420190 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-workload-socket\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:29:06.421278 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.421250 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-credential-socket\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:29:06.421547 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.421516 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-istio-data\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:29:06.422133 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.422111 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-istio-envoy\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:29:06.422932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.422508 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44db3a17-452a-4667-bbc8-0f73b1711233-istiod-ca-cert\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:29:06.430645 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.429089 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44db3a17-452a-4667-bbc8-0f73b1711233-istio-podinfo\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:29:06.439664 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.439567 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bb16a8-hgdpt"] Oct 30 00:29:06.449633 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.449605 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bb16a8-hgdpt"] Oct 30 00:29:06.452563 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.450627 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt9kx\" (UniqueName: \"kubernetes.io/projected/44db3a17-452a-4667-bbc8-0f73b1711233-kube-api-access-jt9kx\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:29:06.456940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.453657 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44db3a17-452a-4667-bbc8-0f73b1711233-istio-token\") pod \"e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:29:06.594933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.592287 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:29:06.919692 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.919612 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a6c9a83-505d-4955-b93f-ec44cf7b5cdb" path="/var/lib/kubelet/pods/9a6c9a83-505d-4955-b93f-ec44cf7b5cdb/volumes" Oct 30 00:29:06.920220 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:06.920205 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0d7bfe5-4113-4a81-8015-936389553f69" path="/var/lib/kubelet/pods/d0d7bfe5-4113-4a81-8015-936389553f69/volumes" Oct 30 00:29:07.044480 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:07.044446 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm"] Oct 30 00:29:07.048871 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:29:07.048845 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44db3a17_452a_4667_bbc8_0f73b1711233.slice/crio-7d4dbc22cee988eda1103923780ee0d88506b4c9ae09e089d970037f77a6fbfd WatchSource:0}: Error finding container 7d4dbc22cee988eda1103923780ee0d88506b4c9ae09e089d970037f77a6fbfd: Status 404 returned error can't find the container with id 7d4dbc22cee988eda1103923780ee0d88506b4c9ae09e089d970037f77a6fbfd Oct 30 00:29:07.052112 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:07.052077 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:29:07.052244 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:07.052143 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:29:07.052244 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:07.052172 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:29:07.400860 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:07.400823 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" event={"ID":"44db3a17-452a-4667-bbc8-0f73b1711233","Type":"ContainerStarted","Data":"a67316a0d7da125cb9b9eaee0e3e791f08ebc0afda6326926f7a9bd19799a8c5"} Oct 30 00:29:07.401102 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:07.400940 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" event={"ID":"44db3a17-452a-4667-bbc8-0f73b1711233","Type":"ContainerStarted","Data":"7d4dbc22cee988eda1103923780ee0d88506b4c9ae09e089d970037f77a6fbfd"} Oct 30 00:29:07.463488 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:07.463436 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" podStartSLOduration=1.46342223 podStartE2EDuration="1.46342223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:29:07.451762274 +0000 UTC m=+1462.657263454" watchObservedRunningTime="2026-10-30 00:29:07.46342223 +0000 UTC m=+1462.668923411" Oct 30 00:29:07.593269 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:07.593233 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:29:08.599968 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:08.597752 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 00:29:08.599968 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:08.597823 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" podUID="44db3a17-452a-4667-bbc8-0f73b1711233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 00:29:09.593095 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:09.593058 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 00:29:09.593266 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:09.593114 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" podUID="44db3a17-452a-4667-bbc8-0f73b1711233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 00:29:10.593448 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:10.593417 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 00:29:10.593819 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:10.593468 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" podUID="44db3a17-452a-4667-bbc8-0f73b1711233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 00:29:11.605936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:11.602332 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 00:29:11.605936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:11.602394 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" podUID="44db3a17-452a-4667-bbc8-0f73b1711233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 00:29:11.722946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:11.721701 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4"] Oct 30 00:29:11.722946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:11.721974 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" containerID="cri-o://7f7d43ea1bbf80ac1fa635633f532414349e8e36c67521bf3696d72bf7b9a9f7" gracePeriod=30 Oct 30 00:29:11.970207 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:11.969973 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:29:12.013340 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:12.013291 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" podUID="27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:29:12.049700 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:12.039994 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:29:12.171347 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:12.171299 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:29:12.188408 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:12.188363 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:29:12.212122 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:12.209797 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:29:12.291003 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:12.290949 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" podUID="27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": context deadline exceeded" Oct 30 00:29:12.324848 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:12.324808 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43dc72-mw75l"] Oct 30 00:29:12.510247 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:12.510200 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43dc72-mw75l" Oct 30 00:29:12.569640 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:12.565300 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:29:12.569640 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:12.566050 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:29:12.569640 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:12.566160 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:29:12.569640 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:12.566271 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:29:12.569640 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:12.568276 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" podUID="ec207720-74f5-4146-93cc-c633e090a5c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.201:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:29:12.621541 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:12.619088 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 00:29:12.621541 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:12.619155 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" podUID="44db3a17-452a-4667-bbc8-0f73b1711233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 00:29:12.621541 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:12.620895 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7ee16fd-d8bc-4aee-b956-62168a6e0fdb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-43dc72-mw75l\" (UID: \"e7ee16fd-d8bc-4aee-b956-62168a6e0fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43dc72-mw75l" Oct 30 00:29:12.621541 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:12.621165 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pptp\" (UniqueName: \"kubernetes.io/projected/e7ee16fd-d8bc-4aee-b956-62168a6e0fdb-kube-api-access-9pptp\") pod \"maas-api-revoke-keys-e2e-api-b-43dc72-mw75l\" (UID: \"e7ee16fd-d8bc-4aee-b956-62168a6e0fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43dc72-mw75l" Oct 30 00:29:12.621541 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:12.621311 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7ee16fd-d8bc-4aee-b956-62168a6e0fdb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-43dc72-mw75l\" (UID: \"e7ee16fd-d8bc-4aee-b956-62168a6e0fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43dc72-mw75l" Oct 30 00:29:12.676573 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:12.657077 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:29:12.685019 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:12.684986 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43dc72-mw75l"] Oct 30 00:29:12.723686 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:12.723628 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7ee16fd-d8bc-4aee-b956-62168a6e0fdb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-43dc72-mw75l\" (UID: \"e7ee16fd-d8bc-4aee-b956-62168a6e0fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43dc72-mw75l" Oct 30 00:29:12.724211 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:12.724187 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pptp\" (UniqueName: \"kubernetes.io/projected/e7ee16fd-d8bc-4aee-b956-62168a6e0fdb-kube-api-access-9pptp\") pod \"maas-api-revoke-keys-e2e-api-b-43dc72-mw75l\" (UID: \"e7ee16fd-d8bc-4aee-b956-62168a6e0fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43dc72-mw75l" Oct 30 00:29:12.725583 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:12.724429 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7ee16fd-d8bc-4aee-b956-62168a6e0fdb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-43dc72-mw75l\" (UID: \"e7ee16fd-d8bc-4aee-b956-62168a6e0fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43dc72-mw75l" Oct 30 00:29:12.725884 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:12.725865 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7ee16fd-d8bc-4aee-b956-62168a6e0fdb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-43dc72-mw75l\" (UID: \"e7ee16fd-d8bc-4aee-b956-62168a6e0fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43dc72-mw75l" Oct 30 00:29:12.744345 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:12.744283 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7ee16fd-d8bc-4aee-b956-62168a6e0fdb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-43dc72-mw75l\" (UID: \"e7ee16fd-d8bc-4aee-b956-62168a6e0fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43dc72-mw75l" Oct 30 00:29:12.749904 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:12.749837 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pptp\" (UniqueName: \"kubernetes.io/projected/e7ee16fd-d8bc-4aee-b956-62168a6e0fdb-kube-api-access-9pptp\") pod \"maas-api-revoke-keys-e2e-api-b-43dc72-mw75l\" (UID: \"e7ee16fd-d8bc-4aee-b956-62168a6e0fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43dc72-mw75l" Oct 30 00:29:13.002903 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:13.002786 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43dc72-mw75l" Oct 30 00:29:13.479417 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:29:13.479383 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7ee16fd_d8bc_4aee_b956_62168a6e0fdb.slice/crio-ea3336c92cf07e47689c780587c4dac9bebe05a0dbed7264ad2508a93b25fd24 WatchSource:0}: Error finding container ea3336c92cf07e47689c780587c4dac9bebe05a0dbed7264ad2508a93b25fd24: Status 404 returned error can't find the container with id ea3336c92cf07e47689c780587c4dac9bebe05a0dbed7264ad2508a93b25fd24 Oct 30 00:29:13.480365 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:13.480336 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43dc72-mw75l"] Oct 30 00:29:13.618138 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:13.608317 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 00:29:13.618138 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:13.608377 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" podUID="44db3a17-452a-4667-bbc8-0f73b1711233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 00:29:13.747934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:13.735824 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43dc72-mw75l" event={"ID":"e7ee16fd-d8bc-4aee-b956-62168a6e0fdb","Type":"ContainerStarted","Data":"ea3336c92cf07e47689c780587c4dac9bebe05a0dbed7264ad2508a93b25fd24"} Oct 30 00:29:14.600206 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:14.596281 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 00:29:14.600206 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:14.596343 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" podUID="44db3a17-452a-4667-bbc8-0f73b1711233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 00:29:14.844312 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:14.844020 5001 patch_prober.go:29] interesting pod/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:29:14.844312 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:14.844091 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:29:14.857397 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:14.855990 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43dc72-mw75l" event={"ID":"e7ee16fd-d8bc-4aee-b956-62168a6e0fdb","Type":"ContainerStarted","Data":"fde3a2a46256e507ee8b8a26271c9a8a2a61e5a9854224981f5a622f00a82105"} Oct 30 00:29:14.905150 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:14.894357 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43dc72-mw75l" podStartSLOduration=2.8943369260000003 podStartE2EDuration="2.894336926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:29:14.893390515 +0000 UTC m=+1470.098891721" watchObservedRunningTime="2026-10-30 00:29:14.894336926 +0000 UTC m=+1470.099838109" Oct 30 00:29:15.007294 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:15.007267 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-20d942-5bgmc"] Oct 30 00:29:15.048975 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:15.047070 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-20d942-5bgmc"] Oct 30 00:29:15.048975 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:15.047194 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-20d942-5bgmc" Oct 30 00:29:15.163102 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:15.163004 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b87b7cf9-722c-471c-8bc0-ffc576905258-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-20d942-5bgmc\" (UID: \"b87b7cf9-722c-471c-8bc0-ffc576905258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-20d942-5bgmc" Oct 30 00:29:15.163287 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:15.163150 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfb2s\" (UniqueName: \"kubernetes.io/projected/b87b7cf9-722c-471c-8bc0-ffc576905258-kube-api-access-wfb2s\") pod \"maas-api-revoke-keys-e2e-praxis-b-20d942-5bgmc\" (UID: \"b87b7cf9-722c-471c-8bc0-ffc576905258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-20d942-5bgmc" Oct 30 00:29:15.163375 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:15.163287 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b87b7cf9-722c-471c-8bc0-ffc576905258-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-20d942-5bgmc\" (UID: \"b87b7cf9-722c-471c-8bc0-ffc576905258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-20d942-5bgmc" Oct 30 00:29:15.280785 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:15.271989 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b87b7cf9-722c-471c-8bc0-ffc576905258-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-20d942-5bgmc\" (UID: \"b87b7cf9-722c-471c-8bc0-ffc576905258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-20d942-5bgmc" Oct 30 00:29:15.280785 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:15.272123 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfb2s\" (UniqueName: \"kubernetes.io/projected/b87b7cf9-722c-471c-8bc0-ffc576905258-kube-api-access-wfb2s\") pod \"maas-api-revoke-keys-e2e-praxis-b-20d942-5bgmc\" (UID: \"b87b7cf9-722c-471c-8bc0-ffc576905258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-20d942-5bgmc" Oct 30 00:29:15.280785 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:15.272296 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b87b7cf9-722c-471c-8bc0-ffc576905258-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-20d942-5bgmc\" (UID: \"b87b7cf9-722c-471c-8bc0-ffc576905258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-20d942-5bgmc" Oct 30 00:29:15.308243 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:15.307050 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b87b7cf9-722c-471c-8bc0-ffc576905258-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-20d942-5bgmc\" (UID: \"b87b7cf9-722c-471c-8bc0-ffc576905258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-20d942-5bgmc" Oct 30 00:29:15.311537 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:15.311503 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b87b7cf9-722c-471c-8bc0-ffc576905258-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-20d942-5bgmc\" (UID: \"b87b7cf9-722c-471c-8bc0-ffc576905258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-20d942-5bgmc" Oct 30 00:29:15.356935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:15.336657 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfb2s\" (UniqueName: \"kubernetes.io/projected/b87b7cf9-722c-471c-8bc0-ffc576905258-kube-api-access-wfb2s\") pod \"maas-api-revoke-keys-e2e-praxis-b-20d942-5bgmc\" (UID: \"b87b7cf9-722c-471c-8bc0-ffc576905258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-20d942-5bgmc" Oct 30 00:29:15.375097 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:15.373671 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-20d942-5bgmc" Oct 30 00:29:15.733184 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:29:15.733135 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb87b7cf9_722c_471c_8bc0_ffc576905258.slice/crio-f493b42b4e3d6c7aa41caa33df92023bccc30051304ea52e95128107739477c3 WatchSource:0}: Error finding container f493b42b4e3d6c7aa41caa33df92023bccc30051304ea52e95128107739477c3: Status 404 returned error can't find the container with id f493b42b4e3d6c7aa41caa33df92023bccc30051304ea52e95128107739477c3 Oct 30 00:29:15.741854 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:15.741828 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-20d942-5bgmc"] Oct 30 00:29:15.836299 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:15.829214 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:29:15.836299 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:15.830845 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:29:15.837327 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:15.837280 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:29:15.868633 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:15.868596 5001 generic.go:356] "Generic (PLEG): container finished" podID="e7ee16fd-d8bc-4aee-b956-62168a6e0fdb" containerID="fde3a2a46256e507ee8b8a26271c9a8a2a61e5a9854224981f5a622f00a82105" exitCode=0 Oct 30 00:29:15.869090 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:15.868672 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43dc72-mw75l" event={"ID":"e7ee16fd-d8bc-4aee-b956-62168a6e0fdb","Type":"ContainerDied","Data":"fde3a2a46256e507ee8b8a26271c9a8a2a61e5a9854224981f5a622f00a82105"} Oct 30 00:29:15.870523 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:15.870485 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-20d942-5bgmc" event={"ID":"b87b7cf9-722c-471c-8bc0-ffc576905258","Type":"ContainerStarted","Data":"f493b42b4e3d6c7aa41caa33df92023bccc30051304ea52e95128107739477c3"} Oct 30 00:29:17.457958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.457903 5001 generic.go:356] "Generic (PLEG): container finished" podID="b87b7cf9-722c-471c-8bc0-ffc576905258" containerID="53d9777a34b3bc557e08469f2f9aa5b0280b67fd22676f5867d1219462aa3f6c" exitCode=0 Oct 30 00:29:17.484106 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.483982 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-20d942-5bgmc" event={"ID":"b87b7cf9-722c-471c-8bc0-ffc576905258","Type":"ContainerDied","Data":"53d9777a34b3bc557e08469f2f9aa5b0280b67fd22676f5867d1219462aa3f6c"} Oct 30 00:29:17.484106 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.484048 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-00df5f-jjgpb"] Oct 30 00:29:17.484106 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.484077 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-00df5f-jjgpb"] Oct 30 00:29:17.536731 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:29:17.536633 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:29:17.782428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.782406 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:29:17.847547 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.847515 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43dc72-mw75l" Oct 30 00:29:17.861837 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.861804 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-credential-socket\") pod \"0c35f644-80ac-439f-9efd-631b26b20a0c\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " Oct 30 00:29:17.862032 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.861875 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-workload-socket\") pod \"0c35f644-80ac-439f-9efd-631b26b20a0c\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " Oct 30 00:29:17.862032 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.861986 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c35f644-80ac-439f-9efd-631b26b20a0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c35f644-80ac-439f-9efd-631b26b20a0c-istio-podinfo\") pod \"0c35f644-80ac-439f-9efd-631b26b20a0c\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " Oct 30 00:29:17.862166 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.862047 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c35f644-80ac-439f-9efd-631b26b20a0c-istio-token\" (UniqueName: \"kubernetes.io/projected/0c35f644-80ac-439f-9efd-631b26b20a0c-istio-token\") pod \"0c35f644-80ac-439f-9efd-631b26b20a0c\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " Oct 30 00:29:17.862166 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.862103 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-workload-certs\") pod \"0c35f644-80ac-439f-9efd-631b26b20a0c\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " Oct 30 00:29:17.862166 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.862132 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-credential-socket" pod "0c35f644-80ac-439f-9efd-631b26b20a0c" (UID: "0c35f644-80ac-439f-9efd-631b26b20a0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:29:17.862371 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.862214 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c35f644-80ac-439f-9efd-631b26b20a0c-kube-api-access-d8vrn\" (UniqueName: \"kubernetes.io/projected/0c35f644-80ac-439f-9efd-631b26b20a0c-kube-api-access-d8vrn\") pod \"0c35f644-80ac-439f-9efd-631b26b20a0c\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " Oct 30 00:29:17.862371 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.862237 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-workload-socket" pod "0c35f644-80ac-439f-9efd-631b26b20a0c" (UID: "0c35f644-80ac-439f-9efd-631b26b20a0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:29:17.862371 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.862339 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-istio-data\") pod \"0c35f644-80ac-439f-9efd-631b26b20a0c\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " Oct 30 00:29:17.862544 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.862376 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-istio-envoy\") pod \"0c35f644-80ac-439f-9efd-631b26b20a0c\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " Oct 30 00:29:17.862544 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.862409 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-workload-certs" pod "0c35f644-80ac-439f-9efd-631b26b20a0c" (UID: "0c35f644-80ac-439f-9efd-631b26b20a0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:29:17.862544 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.862419 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c35f644-80ac-439f-9efd-631b26b20a0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c35f644-80ac-439f-9efd-631b26b20a0c-istiod-ca-cert\") pod \"0c35f644-80ac-439f-9efd-631b26b20a0c\" (UID: \"0c35f644-80ac-439f-9efd-631b26b20a0c\") " Oct 30 00:29:17.862690 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.862575 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-istio-data" pod "0c35f644-80ac-439f-9efd-631b26b20a0c" (UID: "0c35f644-80ac-439f-9efd-631b26b20a0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:29:17.863072 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.863043 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c35f644-80ac-439f-9efd-631b26b20a0c-istiod-ca-cert" pod "0c35f644-80ac-439f-9efd-631b26b20a0c" (UID: "0c35f644-80ac-439f-9efd-631b26b20a0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:29:17.863179 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.863106 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:17.863179 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.863127 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:17.863179 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.863140 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:17.863179 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.863153 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:17.864862 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.864840 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c35f644-80ac-439f-9efd-631b26b20a0c-istio-token" pod "0c35f644-80ac-439f-9efd-631b26b20a0c" (UID: "0c35f644-80ac-439f-9efd-631b26b20a0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:29:17.865070 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.864993 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c35f644-80ac-439f-9efd-631b26b20a0c-kube-api-access-d8vrn" pod "0c35f644-80ac-439f-9efd-631b26b20a0c" (UID: "0c35f644-80ac-439f-9efd-631b26b20a0c"). InnerVolumeSpecName "kube-api-access-d8vrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:29:17.865070 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.865048 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-istio-envoy" pod "0c35f644-80ac-439f-9efd-631b26b20a0c" (UID: "0c35f644-80ac-439f-9efd-631b26b20a0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:29:17.865151 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.865112 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c35f644-80ac-439f-9efd-631b26b20a0c-istio-podinfo" pod "0c35f644-80ac-439f-9efd-631b26b20a0c" (UID: "0c35f644-80ac-439f-9efd-631b26b20a0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:29:17.964246 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.964214 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7ee16fd-d8bc-4aee-b956-62168a6e0fdb-kube-api-access-9pptp\" (UniqueName: \"kubernetes.io/projected/e7ee16fd-d8bc-4aee-b956-62168a6e0fdb-kube-api-access-9pptp\") pod \"e7ee16fd-d8bc-4aee-b956-62168a6e0fdb\" (UID: \"e7ee16fd-d8bc-4aee-b956-62168a6e0fdb\") " Oct 30 00:29:17.964438 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.964419 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7ee16fd-d8bc-4aee-b956-62168a6e0fdb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7ee16fd-d8bc-4aee-b956-62168a6e0fdb-maas-api-xks-ca\") pod \"e7ee16fd-d8bc-4aee-b956-62168a6e0fdb\" (UID: \"e7ee16fd-d8bc-4aee-b956-62168a6e0fdb\") " Oct 30 00:29:17.964558 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.964541 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7ee16fd-d8bc-4aee-b956-62168a6e0fdb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7ee16fd-d8bc-4aee-b956-62168a6e0fdb-maas-api-openshift-ca\") pod \"e7ee16fd-d8bc-4aee-b956-62168a6e0fdb\" (UID: \"e7ee16fd-d8bc-4aee-b956-62168a6e0fdb\") " Oct 30 00:29:17.964987 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.964956 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7ee16fd-d8bc-4aee-b956-62168a6e0fdb-maas-api-openshift-ca" pod "e7ee16fd-d8bc-4aee-b956-62168a6e0fdb" (UID: "e7ee16fd-d8bc-4aee-b956-62168a6e0fdb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:29:17.965556 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.965534 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c35f644-80ac-439f-9efd-631b26b20a0c-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:17.965723 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.965698 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7ee16fd-d8bc-4aee-b956-62168a6e0fdb-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:17.965832 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.965726 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c35f644-80ac-439f-9efd-631b26b20a0c-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:17.965832 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.965742 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8vrn\" (UniqueName: \"kubernetes.io/projected/0c35f644-80ac-439f-9efd-631b26b20a0c-kube-api-access-d8vrn\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:17.965832 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.965756 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c35f644-80ac-439f-9efd-631b26b20a0c-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:17.965832 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.965770 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c35f644-80ac-439f-9efd-631b26b20a0c-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:17.966661 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.966632 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7ee16fd-d8bc-4aee-b956-62168a6e0fdb-kube-api-access-9pptp" pod "e7ee16fd-d8bc-4aee-b956-62168a6e0fdb" (UID: "e7ee16fd-d8bc-4aee-b956-62168a6e0fdb"). InnerVolumeSpecName "kube-api-access-9pptp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:29:17.966661 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:17.966648 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7ee16fd-d8bc-4aee-b956-62168a6e0fdb-maas-api-xks-ca" pod "e7ee16fd-d8bc-4aee-b956-62168a6e0fdb" (UID: "e7ee16fd-d8bc-4aee-b956-62168a6e0fdb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:29:18.067711 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:18.067675 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7ee16fd-d8bc-4aee-b956-62168a6e0fdb-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:18.067711 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:18.067710 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pptp\" (UniqueName: \"kubernetes.io/projected/e7ee16fd-d8bc-4aee-b956-62168a6e0fdb-kube-api-access-9pptp\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:18.469823 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:18.469730 5001 generic.go:356] "Generic (PLEG): container finished" podID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerID="7f7d43ea1bbf80ac1fa635633f532414349e8e36c67521bf3696d72bf7b9a9f7" exitCode=0 Oct 30 00:29:18.469823 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:18.469783 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" Oct 30 00:29:18.469823 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:18.469811 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" event={"ID":"0c35f644-80ac-439f-9efd-631b26b20a0c","Type":"ContainerDied","Data":"7f7d43ea1bbf80ac1fa635633f532414349e8e36c67521bf3696d72bf7b9a9f7"} Oct 30 00:29:18.470392 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:18.469880 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4" event={"ID":"0c35f644-80ac-439f-9efd-631b26b20a0c","Type":"ContainerDied","Data":"b33358192501ebcb0160abf97d1494f6dc0e0b1aebb00f82881335dbd9a6a047"} Oct 30 00:29:18.470392 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:18.469922 5001 scope.go:122] "RemoveContainer" containerID="7f7d43ea1bbf80ac1fa635633f532414349e8e36c67521bf3696d72bf7b9a9f7" Oct 30 00:29:18.472120 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:18.472092 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43dc72-mw75l" event={"ID":"e7ee16fd-d8bc-4aee-b956-62168a6e0fdb","Type":"ContainerDied","Data":"ea3336c92cf07e47689c780587c4dac9bebe05a0dbed7264ad2508a93b25fd24"} Oct 30 00:29:18.472120 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:18.472120 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43dc72-mw75l" Oct 30 00:29:18.472313 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:18.472143 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea3336c92cf07e47689c780587c4dac9bebe05a0dbed7264ad2508a93b25fd24" Oct 30 00:29:18.489150 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:18.489127 5001 scope.go:122] "RemoveContainer" containerID="eee8c4d26b92ea429508ab9169296ac584476f24181e6e3c79ae36f85e5a2b3d" Oct 30 00:29:18.503015 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:18.502937 5001 scope.go:122] "RemoveContainer" containerID="7f7d43ea1bbf80ac1fa635633f532414349e8e36c67521bf3696d72bf7b9a9f7" Oct 30 00:29:18.503336 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:29:18.503304 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f7d43ea1bbf80ac1fa635633f532414349e8e36c67521bf3696d72bf7b9a9f7\": container with ID starting with 7f7d43ea1bbf80ac1fa635633f532414349e8e36c67521bf3696d72bf7b9a9f7 not found: ID does not exist" containerID="7f7d43ea1bbf80ac1fa635633f532414349e8e36c67521bf3696d72bf7b9a9f7" Oct 30 00:29:18.503456 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:18.503341 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f7d43ea1bbf80ac1fa635633f532414349e8e36c67521bf3696d72bf7b9a9f7"} err="failed to get container status \"7f7d43ea1bbf80ac1fa635633f532414349e8e36c67521bf3696d72bf7b9a9f7\": rpc error: code = NotFound desc = could not find container \"7f7d43ea1bbf80ac1fa635633f532414349e8e36c67521bf3696d72bf7b9a9f7\": container with ID starting with 7f7d43ea1bbf80ac1fa635633f532414349e8e36c67521bf3696d72bf7b9a9f7 not found: ID does not exist" Oct 30 00:29:18.503456 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:18.503367 5001 scope.go:122] "RemoveContainer" containerID="eee8c4d26b92ea429508ab9169296ac584476f24181e6e3c79ae36f85e5a2b3d" Oct 30 00:29:18.503682 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:29:18.503656 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eee8c4d26b92ea429508ab9169296ac584476f24181e6e3c79ae36f85e5a2b3d\": container with ID starting with eee8c4d26b92ea429508ab9169296ac584476f24181e6e3c79ae36f85e5a2b3d not found: ID does not exist" containerID="eee8c4d26b92ea429508ab9169296ac584476f24181e6e3c79ae36f85e5a2b3d" Oct 30 00:29:18.503759 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:18.503688 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee8c4d26b92ea429508ab9169296ac584476f24181e6e3c79ae36f85e5a2b3d"} err="failed to get container status \"eee8c4d26b92ea429508ab9169296ac584476f24181e6e3c79ae36f85e5a2b3d\": rpc error: code = NotFound desc = could not find container \"eee8c4d26b92ea429508ab9169296ac584476f24181e6e3c79ae36f85e5a2b3d\": container with ID starting with eee8c4d26b92ea429508ab9169296ac584476f24181e6e3c79ae36f85e5a2b3d not found: ID does not exist" Oct 30 00:29:18.521799 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:18.521764 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4"] Oct 30 00:29:18.531232 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:18.531201 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-6cfcf7-openshift-default-6966bf9459-sj5x4"] Oct 30 00:29:18.648882 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:18.648856 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-20d942-5bgmc" Oct 30 00:29:18.819324 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:18.788870 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b87b7cf9-722c-471c-8bc0-ffc576905258-kube-api-access-wfb2s\" (UniqueName: \"kubernetes.io/projected/b87b7cf9-722c-471c-8bc0-ffc576905258-kube-api-access-wfb2s\") pod \"b87b7cf9-722c-471c-8bc0-ffc576905258\" (UID: \"b87b7cf9-722c-471c-8bc0-ffc576905258\") " Oct 30 00:29:18.820304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:18.820276 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b87b7cf9-722c-471c-8bc0-ffc576905258-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b87b7cf9-722c-471c-8bc0-ffc576905258-maas-api-openshift-ca\") pod \"b87b7cf9-722c-471c-8bc0-ffc576905258\" (UID: \"b87b7cf9-722c-471c-8bc0-ffc576905258\") " Oct 30 00:29:18.820462 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:18.820369 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b87b7cf9-722c-471c-8bc0-ffc576905258-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b87b7cf9-722c-471c-8bc0-ffc576905258-maas-api-xks-ca\") pod \"b87b7cf9-722c-471c-8bc0-ffc576905258\" (UID: \"b87b7cf9-722c-471c-8bc0-ffc576905258\") " Oct 30 00:29:18.837054 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:18.823222 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b87b7cf9-722c-471c-8bc0-ffc576905258-maas-api-openshift-ca" pod "b87b7cf9-722c-471c-8bc0-ffc576905258" (UID: "b87b7cf9-722c-471c-8bc0-ffc576905258"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:29:18.837054 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:18.823645 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b87b7cf9-722c-471c-8bc0-ffc576905258-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:18.837054 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:18.824225 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b87b7cf9-722c-471c-8bc0-ffc576905258-kube-api-access-wfb2s" pod "b87b7cf9-722c-471c-8bc0-ffc576905258" (UID: "b87b7cf9-722c-471c-8bc0-ffc576905258"). InnerVolumeSpecName "kube-api-access-wfb2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:29:18.837054 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:18.825382 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b87b7cf9-722c-471c-8bc0-ffc576905258-maas-api-xks-ca" pod "b87b7cf9-722c-471c-8bc0-ffc576905258" (UID: "b87b7cf9-722c-471c-8bc0-ffc576905258"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:29:18.925480 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:18.925437 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" path="/var/lib/kubelet/pods/0c35f644-80ac-439f-9efd-631b26b20a0c/volumes" Oct 30 00:29:18.925660 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:18.925448 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b87b7cf9-722c-471c-8bc0-ffc576905258-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:18.925660 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:18.925507 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfb2s\" (UniqueName: \"kubernetes.io/projected/b87b7cf9-722c-471c-8bc0-ffc576905258-kube-api-access-wfb2s\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:18.926094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:18.926071 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5046332-5ea3-4746-8100-f28a12a95aee" path="/var/lib/kubelet/pods/b5046332-5ea3-4746-8100-f28a12a95aee/volumes" Oct 30 00:29:19.353085 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:19.353039 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:29:19.353288 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:19.353113 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" podUID="242dd865-71a7-4ade-a9ae-e80adc01cf59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:29:19.482982 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:19.482947 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-20d942-5bgmc" event={"ID":"b87b7cf9-722c-471c-8bc0-ffc576905258","Type":"ContainerDied","Data":"f493b42b4e3d6c7aa41caa33df92023bccc30051304ea52e95128107739477c3"} Oct 30 00:29:19.483404 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:19.483023 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f493b42b4e3d6c7aa41caa33df92023bccc30051304ea52e95128107739477c3" Oct 30 00:29:19.483404 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:19.482964 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-20d942-5bgmc" Oct 30 00:29:21.504731 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:21.504634 5001 generic.go:356] "Generic (PLEG): container finished" podID="242dd865-71a7-4ade-a9ae-e80adc01cf59" containerID="3ca7cb7aa17eff4d5d03a35e8762ef1fc12a47cab61561c39e84758377dee015" exitCode=0 Oct 30 00:29:21.504731 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:21.504709 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" event={"ID":"242dd865-71a7-4ade-a9ae-e80adc01cf59","Type":"ContainerDied","Data":"3ca7cb7aa17eff4d5d03a35e8762ef1fc12a47cab61561c39e84758377dee015"} Oct 30 00:29:21.505219 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:21.505152 5001 scope.go:122] "RemoveContainer" containerID="3ca7cb7aa17eff4d5d03a35e8762ef1fc12a47cab61561c39e84758377dee015" Oct 30 00:29:21.507002 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:21.506973 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:29:21.507097 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:21.507056 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:29:21.507097 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:21.507089 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:29:22.543991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:22.543952 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" event={"ID":"242dd865-71a7-4ade-a9ae-e80adc01cf59","Type":"ContainerStarted","Data":"5685b64fd757df5c4a8346e65c88697a2632d6a052a4faf76e483b3e0ed92539"} Oct 30 00:29:22.844887 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:22.844775 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-43dc72-5f4bd85fdc-2fn22"] Oct 30 00:29:23.160005 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:23.159434 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:29:23.160005 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:23.159560 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 00:29:23.160005 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:23.159622 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" podUID="242dd865-71a7-4ade-a9ae-e80adc01cf59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 00:29:23.712062 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:23.711998 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4500d5-x6w85"] Oct 30 00:29:23.829936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:23.829688 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4500d5-x6w85"] Oct 30 00:29:24.111852 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:24.111801 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:29:24.134208 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:24.134136 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:29:24.153281 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:24.149109 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 00:29:24.153613 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:24.153580 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" podUID="242dd865-71a7-4ade-a9ae-e80adc01cf59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 00:29:24.188554 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:24.188518 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:29:24.956283 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:24.956246 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21ee8ab8-ee55-4801-a39e-8d5ad1f596ee" path="/var/lib/kubelet/pods/21ee8ab8-ee55-4801-a39e-8d5ad1f596ee/volumes" Oct 30 00:29:24.970540 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:24.970507 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl"] Oct 30 00:29:24.970778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:24.970748 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" podUID="a684c9be-f1fe-420b-a3a0-16ec60772131" containerName="istio-proxy" containerID="cri-o://99b9fd83eeb52926a660a6e40e04bacc5ff3cabe8fcfc9b73ecae525375b802e" gracePeriod=30 Oct 30 00:29:25.150549 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:25.150490 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 00:29:25.150852 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:25.150809 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" podUID="242dd865-71a7-4ade-a9ae-e80adc01cf59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 00:29:25.546216 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:25.546174 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-20d942-9ddb98f76-2bg4x"] Oct 30 00:29:26.146377 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:26.146343 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 00:29:26.146806 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:26.146398 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" podUID="242dd865-71a7-4ade-a9ae-e80adc01cf59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 00:29:27.147937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:27.146699 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 00:29:27.147937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:27.146754 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" podUID="242dd865-71a7-4ade-a9ae-e80adc01cf59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 00:29:27.847264 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:27.845803 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-43dc72-5f4bd85fdc-2fn22" podUID="6c04baf6-1ee8-47ee-b3a7-3f92393b644b" containerName="payload-pre-processing" containerID="cri-o://61ff95047d6a666808973edaa21c3a1550c167ac068dc6a34f148966f060b52a" gracePeriod=25 Oct 30 00:29:28.146553 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:28.146461 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 00:29:28.146703 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:28.146561 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" podUID="242dd865-71a7-4ade-a9ae-e80adc01cf59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 00:29:28.193071 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:28.193045 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-43dc72-5f4bd85fdc-2fn22" Oct 30 00:29:28.290253 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:28.290215 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c04baf6-1ee8-47ee-b3a7-3f92393b644b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c04baf6-1ee8-47ee-b3a7-3f92393b644b-plugins-config-volume\") pod \"6c04baf6-1ee8-47ee-b3a7-3f92393b644b\" (UID: \"6c04baf6-1ee8-47ee-b3a7-3f92393b644b\") " Oct 30 00:29:28.290447 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:28.290308 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c04baf6-1ee8-47ee-b3a7-3f92393b644b-kube-api-access-ddhg2\" (UniqueName: \"kubernetes.io/projected/6c04baf6-1ee8-47ee-b3a7-3f92393b644b-kube-api-access-ddhg2\") pod \"6c04baf6-1ee8-47ee-b3a7-3f92393b644b\" (UID: \"6c04baf6-1ee8-47ee-b3a7-3f92393b644b\") " Oct 30 00:29:28.290590 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:28.290565 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c04baf6-1ee8-47ee-b3a7-3f92393b644b-plugins-config-volume" pod "6c04baf6-1ee8-47ee-b3a7-3f92393b644b" (UID: "6c04baf6-1ee8-47ee-b3a7-3f92393b644b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:29:28.290937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:28.290907 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c04baf6-1ee8-47ee-b3a7-3f92393b644b-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:28.292525 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:28.292501 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c04baf6-1ee8-47ee-b3a7-3f92393b644b-kube-api-access-ddhg2" pod "6c04baf6-1ee8-47ee-b3a7-3f92393b644b" (UID: "6c04baf6-1ee8-47ee-b3a7-3f92393b644b"). InnerVolumeSpecName "kube-api-access-ddhg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:29:28.393102 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:28.393061 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddhg2\" (UniqueName: \"kubernetes.io/projected/6c04baf6-1ee8-47ee-b3a7-3f92393b644b-kube-api-access-ddhg2\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:28.916278 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:28.916240 5001 generic.go:356] "Generic (PLEG): container finished" podID="6c04baf6-1ee8-47ee-b3a7-3f92393b644b" containerID="61ff95047d6a666808973edaa21c3a1550c167ac068dc6a34f148966f060b52a" exitCode=0 Oct 30 00:29:28.916626 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:28.916609 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-43dc72-5f4bd85fdc-2fn22" Oct 30 00:29:28.922335 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:28.922308 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-43dc72-5f4bd85fdc-2fn22" event={"ID":"6c04baf6-1ee8-47ee-b3a7-3f92393b644b","Type":"ContainerDied","Data":"61ff95047d6a666808973edaa21c3a1550c167ac068dc6a34f148966f060b52a"} Oct 30 00:29:28.922480 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:28.922357 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-43dc72-5f4bd85fdc-2fn22" event={"ID":"6c04baf6-1ee8-47ee-b3a7-3f92393b644b","Type":"ContainerDied","Data":"b47c117f61967e5f75dfa05f4c849af88091f2a2ecb24f0aca6f54ed6f01a384"} Oct 30 00:29:28.922480 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:28.922388 5001 scope.go:122] "RemoveContainer" containerID="61ff95047d6a666808973edaa21c3a1550c167ac068dc6a34f148966f060b52a" Oct 30 00:29:28.936371 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:28.936347 5001 scope.go:122] "RemoveContainer" containerID="61ff95047d6a666808973edaa21c3a1550c167ac068dc6a34f148966f060b52a" Oct 30 00:29:28.936709 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:29:28.936683 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61ff95047d6a666808973edaa21c3a1550c167ac068dc6a34f148966f060b52a\": container with ID starting with 61ff95047d6a666808973edaa21c3a1550c167ac068dc6a34f148966f060b52a not found: ID does not exist" containerID="61ff95047d6a666808973edaa21c3a1550c167ac068dc6a34f148966f060b52a" Oct 30 00:29:28.936803 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:28.936715 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61ff95047d6a666808973edaa21c3a1550c167ac068dc6a34f148966f060b52a"} err="failed to get container status \"61ff95047d6a666808973edaa21c3a1550c167ac068dc6a34f148966f060b52a\": rpc error: code = NotFound desc = could not find container \"61ff95047d6a666808973edaa21c3a1550c167ac068dc6a34f148966f060b52a\": container with ID starting with 61ff95047d6a666808973edaa21c3a1550c167ac068dc6a34f148966f060b52a not found: ID does not exist" Oct 30 00:29:28.967220 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:28.967182 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-43dc72-5f4bd85fdc-2fn22"] Oct 30 00:29:28.976695 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:28.976664 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-43dc72-5f4bd85fdc-2fn22"] Oct 30 00:29:29.182387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:29.182301 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:29:29.182832 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:29.182796 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:29:29.183795 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:29.183771 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:29:29.275971 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:29.275932 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:29:29.276415 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:29.275994 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" podUID="a684c9be-f1fe-420b-a3a0-16ec60772131" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:29:29.794395 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:29.794359 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-422c9df5-v25fr"] Oct 30 00:29:29.795156 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:29.795136 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e7ee16fd-d8bc-4aee-b956-62168a6e0fdb" containerName="revoke-keys" Oct 30 00:29:29.795156 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:29.795155 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7ee16fd-d8bc-4aee-b956-62168a6e0fdb" containerName="revoke-keys" Oct 30 00:29:29.795332 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:29.795167 5001 cpu_manager.go:414] "containerMap: removing container" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" Oct 30 00:29:29.795332 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:29.795172 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" Oct 30 00:29:29.795332 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:29.795193 5001 cpu_manager.go:414] "containerMap: removing container" podUID="6c04baf6-1ee8-47ee-b3a7-3f92393b644b" containerName="payload-pre-processing" Oct 30 00:29:29.795332 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:29.795199 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c04baf6-1ee8-47ee-b3a7-3f92393b644b" containerName="payload-pre-processing" Oct 30 00:29:29.795332 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:29.795227 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b87b7cf9-722c-471c-8bc0-ffc576905258" containerName="revoke-keys" Oct 30 00:29:29.795332 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:29.795231 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b87b7cf9-722c-471c-8bc0-ffc576905258" containerName="revoke-keys" Oct 30 00:29:29.795332 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:29.795331 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b87b7cf9-722c-471c-8bc0-ffc576905258" containerName="revoke-keys" Oct 30 00:29:29.795552 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:29.795343 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7ee16fd-d8bc-4aee-b956-62168a6e0fdb" containerName="revoke-keys" Oct 30 00:29:29.795552 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:29.795349 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" Oct 30 00:29:29.795552 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:29.795358 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c04baf6-1ee8-47ee-b3a7-3f92393b644b" containerName="payload-pre-processing" Oct 30 00:29:29.800269 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:29.800250 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422c9df5-v25fr" Oct 30 00:29:29.838134 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:29.838110 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:29:29.838248 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:29.838228 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:29:29.841875 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:29.841856 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:29:29.842463 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:29.842442 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:29:29.845527 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:29.845500 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-422c9df5-v25fr"] Oct 30 00:29:29.863290 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:29.863268 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9aa57dc-4f61-4001-8573-9cd8ee69246f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-422c9df5-v25fr\" (UID: \"a9aa57dc-4f61-4001-8573-9cd8ee69246f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422c9df5-v25fr" Oct 30 00:29:29.863394 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:29.863317 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kmw2\" (UniqueName: \"kubernetes.io/projected/a9aa57dc-4f61-4001-8573-9cd8ee69246f-kube-api-access-7kmw2\") pod \"maas-api-revoke-sub-422c9df5-v25fr\" (UID: \"a9aa57dc-4f61-4001-8573-9cd8ee69246f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422c9df5-v25fr" Oct 30 00:29:29.863657 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:29.863635 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9aa57dc-4f61-4001-8573-9cd8ee69246f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-422c9df5-v25fr\" (UID: \"a9aa57dc-4f61-4001-8573-9cd8ee69246f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422c9df5-v25fr" Oct 30 00:29:29.965123 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:29.965090 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9aa57dc-4f61-4001-8573-9cd8ee69246f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-422c9df5-v25fr\" (UID: \"a9aa57dc-4f61-4001-8573-9cd8ee69246f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422c9df5-v25fr" Oct 30 00:29:29.965300 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:29.965209 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9aa57dc-4f61-4001-8573-9cd8ee69246f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-422c9df5-v25fr\" (UID: \"a9aa57dc-4f61-4001-8573-9cd8ee69246f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422c9df5-v25fr" Oct 30 00:29:29.965300 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:29.965242 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kmw2\" (UniqueName: \"kubernetes.io/projected/a9aa57dc-4f61-4001-8573-9cd8ee69246f-kube-api-access-7kmw2\") pod \"maas-api-revoke-sub-422c9df5-v25fr\" (UID: \"a9aa57dc-4f61-4001-8573-9cd8ee69246f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422c9df5-v25fr" Oct 30 00:29:29.965872 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:29.965851 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9aa57dc-4f61-4001-8573-9cd8ee69246f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-422c9df5-v25fr\" (UID: \"a9aa57dc-4f61-4001-8573-9cd8ee69246f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422c9df5-v25fr" Oct 30 00:29:29.967656 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:29.967635 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9aa57dc-4f61-4001-8573-9cd8ee69246f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-422c9df5-v25fr\" (UID: \"a9aa57dc-4f61-4001-8573-9cd8ee69246f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422c9df5-v25fr" Oct 30 00:29:29.986092 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:29.986069 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kmw2\" (UniqueName: \"kubernetes.io/projected/a9aa57dc-4f61-4001-8573-9cd8ee69246f-kube-api-access-7kmw2\") pod \"maas-api-revoke-sub-422c9df5-v25fr\" (UID: \"a9aa57dc-4f61-4001-8573-9cd8ee69246f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422c9df5-v25fr" Oct 30 00:29:30.115095 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:30.115061 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422c9df5-v25fr" Oct 30 00:29:30.259317 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:30.259291 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:29:30.372460 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:30.372373 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a684c9be-f1fe-420b-a3a0-16ec60772131-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a684c9be-f1fe-420b-a3a0-16ec60772131-istio-podinfo\") pod \"a684c9be-f1fe-420b-a3a0-16ec60772131\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " Oct 30 00:29:30.372955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:30.372477 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a684c9be-f1fe-420b-a3a0-16ec60772131-kube-api-access-6kl5h\" (UniqueName: \"kubernetes.io/projected/a684c9be-f1fe-420b-a3a0-16ec60772131-kube-api-access-6kl5h\") pod \"a684c9be-f1fe-420b-a3a0-16ec60772131\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " Oct 30 00:29:30.372955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:30.372536 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-istio-data\") pod \"a684c9be-f1fe-420b-a3a0-16ec60772131\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " Oct 30 00:29:30.372955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:30.372587 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-workload-certs\") pod \"a684c9be-f1fe-420b-a3a0-16ec60772131\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " Oct 30 00:29:30.372955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:30.372671 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-istio-envoy\") pod \"a684c9be-f1fe-420b-a3a0-16ec60772131\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " Oct 30 00:29:30.372955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:30.372709 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-credential-socket\") pod \"a684c9be-f1fe-420b-a3a0-16ec60772131\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " Oct 30 00:29:30.372955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:30.372820 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a684c9be-f1fe-420b-a3a0-16ec60772131-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a684c9be-f1fe-420b-a3a0-16ec60772131-istiod-ca-cert\") pod \"a684c9be-f1fe-420b-a3a0-16ec60772131\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " Oct 30 00:29:30.372955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:30.372843 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-istio-data" pod "a684c9be-f1fe-420b-a3a0-16ec60772131" (UID: "a684c9be-f1fe-420b-a3a0-16ec60772131"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:29:30.372955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:30.372857 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-workload-certs" pod "a684c9be-f1fe-420b-a3a0-16ec60772131" (UID: "a684c9be-f1fe-420b-a3a0-16ec60772131"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:29:30.372955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:30.372885 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-workload-socket\") pod \"a684c9be-f1fe-420b-a3a0-16ec60772131\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " Oct 30 00:29:30.373463 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:30.373015 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a684c9be-f1fe-420b-a3a0-16ec60772131-istio-token\" (UniqueName: \"kubernetes.io/projected/a684c9be-f1fe-420b-a3a0-16ec60772131-istio-token\") pod \"a684c9be-f1fe-420b-a3a0-16ec60772131\" (UID: \"a684c9be-f1fe-420b-a3a0-16ec60772131\") " Oct 30 00:29:30.373463 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:30.373055 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-credential-socket" pod "a684c9be-f1fe-420b-a3a0-16ec60772131" (UID: "a684c9be-f1fe-420b-a3a0-16ec60772131"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:29:30.373463 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:30.373119 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-workload-socket" pod "a684c9be-f1fe-420b-a3a0-16ec60772131" (UID: "a684c9be-f1fe-420b-a3a0-16ec60772131"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:29:30.373463 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:30.373415 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a684c9be-f1fe-420b-a3a0-16ec60772131-istiod-ca-cert" pod "a684c9be-f1fe-420b-a3a0-16ec60772131" (UID: "a684c9be-f1fe-420b-a3a0-16ec60772131"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:29:30.373773 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:30.373756 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:30.373829 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:30.373780 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a684c9be-f1fe-420b-a3a0-16ec60772131-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:30.373829 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:30.373790 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:30.373829 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:30.373799 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:30.373829 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:30.373807 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:30.375329 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:30.375299 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a684c9be-f1fe-420b-a3a0-16ec60772131-istio-podinfo" pod "a684c9be-f1fe-420b-a3a0-16ec60772131" (UID: "a684c9be-f1fe-420b-a3a0-16ec60772131"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:29:30.375619 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:30.375591 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a684c9be-f1fe-420b-a3a0-16ec60772131-istio-token" pod "a684c9be-f1fe-420b-a3a0-16ec60772131" (UID: "a684c9be-f1fe-420b-a3a0-16ec60772131"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:29:30.377751 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:30.377269 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a684c9be-f1fe-420b-a3a0-16ec60772131-kube-api-access-6kl5h" pod "a684c9be-f1fe-420b-a3a0-16ec60772131" (UID: "a684c9be-f1fe-420b-a3a0-16ec60772131"). InnerVolumeSpecName "kube-api-access-6kl5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:29:30.377751 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:30.377390 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-istio-envoy" pod "a684c9be-f1fe-420b-a3a0-16ec60772131" (UID: "a684c9be-f1fe-420b-a3a0-16ec60772131"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:29:30.475639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:30.475606 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a684c9be-f1fe-420b-a3a0-16ec60772131-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:30.475639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:30.475633 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kl5h\" (UniqueName: \"kubernetes.io/projected/a684c9be-f1fe-420b-a3a0-16ec60772131-kube-api-access-6kl5h\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:30.475639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:30.475643 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a684c9be-f1fe-420b-a3a0-16ec60772131-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:30.475873 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:30.475652 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a684c9be-f1fe-420b-a3a0-16ec60772131-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:30.487357 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:29:30.487333 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9aa57dc_4f61_4001_8573_9cd8ee69246f.slice/crio-8b5a1650b5bd9cca3c48241aa6e52998cb71c74befa8add88a1b93ac886542af WatchSource:0}: Error finding container 8b5a1650b5bd9cca3c48241aa6e52998cb71c74befa8add88a1b93ac886542af: Status 404 returned error can't find the container with id 8b5a1650b5bd9cca3c48241aa6e52998cb71c74befa8add88a1b93ac886542af Oct 30 00:29:30.488194 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:30.488170 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-422c9df5-v25fr"] Oct 30 00:29:30.547325 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:30.547288 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-20d942-9ddb98f76-2bg4x" podUID="1b18b95b-c9bb-4019-a6fe-79435057127c" containerName="payload-processing" containerID="cri-o://ead4bab7f69df7e23c858d749c09fd53700719c76a65b7372740f702496df36c" gracePeriod=25 Oct 30 00:29:30.645946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:30.645850 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd"] Oct 30 00:29:31.136094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:31.129581 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c04baf6-1ee8-47ee-b3a7-3f92393b644b" path="/var/lib/kubelet/pods/6c04baf6-1ee8-47ee-b3a7-3f92393b644b/volumes" Oct 30 00:29:31.261028 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:31.260307 5001 generic.go:356] "Generic (PLEG): container finished" podID="a684c9be-f1fe-420b-a3a0-16ec60772131" containerID="99b9fd83eeb52926a660a6e40e04bacc5ff3cabe8fcfc9b73ecae525375b802e" exitCode=0 Oct 30 00:29:31.261028 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:31.260547 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" event={"ID":"a684c9be-f1fe-420b-a3a0-16ec60772131","Type":"ContainerDied","Data":"99b9fd83eeb52926a660a6e40e04bacc5ff3cabe8fcfc9b73ecae525375b802e"} Oct 30 00:29:31.261028 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:31.260573 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" Oct 30 00:29:31.261028 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:31.260600 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl" event={"ID":"a684c9be-f1fe-420b-a3a0-16ec60772131","Type":"ContainerDied","Data":"2c93fa7a05592e1352a687f981d2bca8622191545b10ddd0e87d79e7528b5ee1"} Oct 30 00:29:31.261028 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:31.260630 5001 scope.go:122] "RemoveContainer" containerID="99b9fd83eeb52926a660a6e40e04bacc5ff3cabe8fcfc9b73ecae525375b802e" Oct 30 00:29:31.264609 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:31.263373 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-20d942-9ddb98f76-2bg4x" Oct 30 00:29:31.280148 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:31.279054 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422c9df5-v25fr" event={"ID":"a9aa57dc-4f61-4001-8573-9cd8ee69246f","Type":"ContainerStarted","Data":"8b5a1650b5bd9cca3c48241aa6e52998cb71c74befa8add88a1b93ac886542af"} Oct 30 00:29:31.284118 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:31.282791 5001 generic.go:356] "Generic (PLEG): container finished" podID="1b18b95b-c9bb-4019-a6fe-79435057127c" containerID="ead4bab7f69df7e23c858d749c09fd53700719c76a65b7372740f702496df36c" exitCode=0 Oct 30 00:29:31.284118 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:31.283352 5001 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" secret="" err="secret \"e2e-praxis-b-4500d5-openshift-default-dockercfg-vg6gr\" not found" Oct 30 00:29:31.284118 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:31.283849 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-20d942-9ddb98f76-2bg4x" Oct 30 00:29:31.284118 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:31.284040 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-20d942-9ddb98f76-2bg4x" event={"ID":"1b18b95b-c9bb-4019-a6fe-79435057127c","Type":"ContainerDied","Data":"ead4bab7f69df7e23c858d749c09fd53700719c76a65b7372740f702496df36c"} Oct 30 00:29:31.300526 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:31.299837 5001 scope.go:122] "RemoveContainer" containerID="06ebf0b626bf6de28c6b1d380ef384c11f214f1de1bab6ea1cac5ce013619781" Oct 30 00:29:31.311736 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:31.311711 5001 scope.go:122] "RemoveContainer" containerID="99b9fd83eeb52926a660a6e40e04bacc5ff3cabe8fcfc9b73ecae525375b802e" Oct 30 00:29:31.312045 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:29:31.312026 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99b9fd83eeb52926a660a6e40e04bacc5ff3cabe8fcfc9b73ecae525375b802e\": container with ID starting with 99b9fd83eeb52926a660a6e40e04bacc5ff3cabe8fcfc9b73ecae525375b802e not found: ID does not exist" containerID="99b9fd83eeb52926a660a6e40e04bacc5ff3cabe8fcfc9b73ecae525375b802e" Oct 30 00:29:31.312125 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:31.312053 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99b9fd83eeb52926a660a6e40e04bacc5ff3cabe8fcfc9b73ecae525375b802e"} err="failed to get container status \"99b9fd83eeb52926a660a6e40e04bacc5ff3cabe8fcfc9b73ecae525375b802e\": rpc error: code = NotFound desc = could not find container \"99b9fd83eeb52926a660a6e40e04bacc5ff3cabe8fcfc9b73ecae525375b802e\": container with ID starting with 99b9fd83eeb52926a660a6e40e04bacc5ff3cabe8fcfc9b73ecae525375b802e not found: ID does not exist" Oct 30 00:29:31.312125 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:31.312072 5001 scope.go:122] "RemoveContainer" containerID="06ebf0b626bf6de28c6b1d380ef384c11f214f1de1bab6ea1cac5ce013619781" Oct 30 00:29:31.312322 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:29:31.312301 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06ebf0b626bf6de28c6b1d380ef384c11f214f1de1bab6ea1cac5ce013619781\": container with ID starting with 06ebf0b626bf6de28c6b1d380ef384c11f214f1de1bab6ea1cac5ce013619781 not found: ID does not exist" containerID="06ebf0b626bf6de28c6b1d380ef384c11f214f1de1bab6ea1cac5ce013619781" Oct 30 00:29:31.312360 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:31.312330 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06ebf0b626bf6de28c6b1d380ef384c11f214f1de1bab6ea1cac5ce013619781"} err="failed to get container status \"06ebf0b626bf6de28c6b1d380ef384c11f214f1de1bab6ea1cac5ce013619781\": rpc error: code = NotFound desc = could not find container \"06ebf0b626bf6de28c6b1d380ef384c11f214f1de1bab6ea1cac5ce013619781\": container with ID starting with 06ebf0b626bf6de28c6b1d380ef384c11f214f1de1bab6ea1cac5ce013619781 not found: ID does not exist" Oct 30 00:29:31.312360 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:31.312348 5001 scope.go:122] "RemoveContainer" containerID="ead4bab7f69df7e23c858d749c09fd53700719c76a65b7372740f702496df36c" Oct 30 00:29:31.357279 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:31.357228 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422c9df5-v25fr" podStartSLOduration=2.357216068 podStartE2EDuration="2.357216068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:29:31.356956483 +0000 UTC m=+1486.562457661" watchObservedRunningTime="2026-10-30 00:29:31.357216068 +0000 UTC m=+1486.562717248" Oct 30 00:29:31.391822 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:31.391795 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl"] Oct 30 00:29:31.396192 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:31.396165 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-00df5f-openshift-default-65cc88485-4kxzl"] Oct 30 00:29:31.432384 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:31.432355 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b18b95b-c9bb-4019-a6fe-79435057127c-kube-api-access-g7vbn\" (UniqueName: \"kubernetes.io/projected/1b18b95b-c9bb-4019-a6fe-79435057127c-kube-api-access-g7vbn\") pod \"1b18b95b-c9bb-4019-a6fe-79435057127c\" (UID: \"1b18b95b-c9bb-4019-a6fe-79435057127c\") " Oct 30 00:29:31.432517 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:31.432444 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b18b95b-c9bb-4019-a6fe-79435057127c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b18b95b-c9bb-4019-a6fe-79435057127c-plugins-config-volume\") pod \"1b18b95b-c9bb-4019-a6fe-79435057127c\" (UID: \"1b18b95b-c9bb-4019-a6fe-79435057127c\") " Oct 30 00:29:31.432865 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:31.432840 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b18b95b-c9bb-4019-a6fe-79435057127c-plugins-config-volume" pod "1b18b95b-c9bb-4019-a6fe-79435057127c" (UID: "1b18b95b-c9bb-4019-a6fe-79435057127c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:29:31.433645 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:31.433626 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b18b95b-c9bb-4019-a6fe-79435057127c-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:31.434410 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:31.434394 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b18b95b-c9bb-4019-a6fe-79435057127c-kube-api-access-g7vbn" pod "1b18b95b-c9bb-4019-a6fe-79435057127c" (UID: "1b18b95b-c9bb-4019-a6fe-79435057127c"). InnerVolumeSpecName "kube-api-access-g7vbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:29:31.534824 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:31.534751 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7vbn\" (UniqueName: \"kubernetes.io/projected/1b18b95b-c9bb-4019-a6fe-79435057127c-kube-api-access-g7vbn\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:31.632685 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:31.632656 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-20d942-9ddb98f76-2bg4x"] Oct 30 00:29:31.641212 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:31.641188 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-20d942-9ddb98f76-2bg4x"] Oct 30 00:29:32.309786 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:32.309748 5001 generic.go:356] "Generic (PLEG): container finished" podID="a9aa57dc-4f61-4001-8573-9cd8ee69246f" containerID="df59fad72286f3a4f28ab5fd93f26d9b390383b8e4ebb66ffd7da0fe40a551bc" exitCode=0 Oct 30 00:29:32.309989 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:32.309885 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422c9df5-v25fr" event={"ID":"a9aa57dc-4f61-4001-8573-9cd8ee69246f","Type":"ContainerDied","Data":"df59fad72286f3a4f28ab5fd93f26d9b390383b8e4ebb66ffd7da0fe40a551bc"} Oct 30 00:29:32.312381 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:32.312354 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" podUID="242dd865-71a7-4ade-a9ae-e80adc01cf59" containerName="istio-proxy" containerID="cri-o://5685b64fd757df5c4a8346e65c88697a2632d6a052a4faf76e483b3e0ed92539" gracePeriod=30 Oct 30 00:29:32.441761 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:32.441712 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-5658554c47-sq9k7"] Oct 30 00:29:32.442661 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:32.442626 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-5658554c47-sq9k7" podUID="17582778-cc63-4b44-ac34-ee9a4c866bba" containerName="payload-pre-processing" containerID="cri-o://827f8f579ad97a31a23a6e223c696bf65f534a64bf1811da87bec5389984bd59" gracePeriod=30 Oct 30 00:29:33.215181 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:33.215143 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b18b95b-c9bb-4019-a6fe-79435057127c" path="/var/lib/kubelet/pods/1b18b95b-c9bb-4019-a6fe-79435057127c/volumes" Oct 30 00:29:33.215719 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:33.215699 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a684c9be-f1fe-420b-a3a0-16ec60772131" path="/var/lib/kubelet/pods/a684c9be-f1fe-420b-a3a0-16ec60772131/volumes" Oct 30 00:29:33.216109 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:33.216092 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a15230-kt696"] Oct 30 00:29:33.216202 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:33.216126 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a15230-kt696"] Oct 30 00:29:33.227303 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:33.227273 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-5658554c47-sq9k7" Oct 30 00:29:33.323671 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:33.323625 5001 generic.go:356] "Generic (PLEG): container finished" podID="17582778-cc63-4b44-ac34-ee9a4c866bba" containerID="827f8f579ad97a31a23a6e223c696bf65f534a64bf1811da87bec5389984bd59" exitCode=0 Oct 30 00:29:33.323671 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:33.323666 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-5658554c47-sq9k7" Oct 30 00:29:33.323949 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:33.323703 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-5658554c47-sq9k7" event={"ID":"17582778-cc63-4b44-ac34-ee9a4c866bba","Type":"ContainerDied","Data":"827f8f579ad97a31a23a6e223c696bf65f534a64bf1811da87bec5389984bd59"} Oct 30 00:29:33.323949 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:33.323757 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-5658554c47-sq9k7" event={"ID":"17582778-cc63-4b44-ac34-ee9a4c866bba","Type":"ContainerDied","Data":"624fc06b78a59d156542cfd992d14d600793f458e8eaef82345bc54a2ec697da"} Oct 30 00:29:33.323949 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:33.323783 5001 scope.go:122] "RemoveContainer" containerID="827f8f579ad97a31a23a6e223c696bf65f534a64bf1811da87bec5389984bd59" Oct 30 00:29:33.326358 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:33.326330 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17582778-cc63-4b44-ac34-ee9a4c866bba-kube-api-access-jmrvj\" (UniqueName: \"kubernetes.io/projected/17582778-cc63-4b44-ac34-ee9a4c866bba-kube-api-access-jmrvj\") pod \"17582778-cc63-4b44-ac34-ee9a4c866bba\" (UID: \"17582778-cc63-4b44-ac34-ee9a4c866bba\") " Oct 30 00:29:33.326678 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:33.326655 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17582778-cc63-4b44-ac34-ee9a4c866bba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17582778-cc63-4b44-ac34-ee9a4c866bba-plugins-config-volume\") pod \"17582778-cc63-4b44-ac34-ee9a4c866bba\" (UID: \"17582778-cc63-4b44-ac34-ee9a4c866bba\") " Oct 30 00:29:33.327144 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:33.327115 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17582778-cc63-4b44-ac34-ee9a4c866bba-plugins-config-volume" pod "17582778-cc63-4b44-ac34-ee9a4c866bba" (UID: "17582778-cc63-4b44-ac34-ee9a4c866bba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:29:33.327439 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:33.327421 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17582778-cc63-4b44-ac34-ee9a4c866bba-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:33.328718 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:33.328691 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17582778-cc63-4b44-ac34-ee9a4c866bba-kube-api-access-jmrvj" pod "17582778-cc63-4b44-ac34-ee9a4c866bba" (UID: "17582778-cc63-4b44-ac34-ee9a4c866bba"). InnerVolumeSpecName "kube-api-access-jmrvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:29:33.337098 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:33.337078 5001 scope.go:122] "RemoveContainer" containerID="827f8f579ad97a31a23a6e223c696bf65f534a64bf1811da87bec5389984bd59" Oct 30 00:29:33.337382 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:29:33.337362 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"827f8f579ad97a31a23a6e223c696bf65f534a64bf1811da87bec5389984bd59\": container with ID starting with 827f8f579ad97a31a23a6e223c696bf65f534a64bf1811da87bec5389984bd59 not found: ID does not exist" containerID="827f8f579ad97a31a23a6e223c696bf65f534a64bf1811da87bec5389984bd59" Oct 30 00:29:33.337453 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:33.337389 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"827f8f579ad97a31a23a6e223c696bf65f534a64bf1811da87bec5389984bd59"} err="failed to get container status \"827f8f579ad97a31a23a6e223c696bf65f534a64bf1811da87bec5389984bd59\": rpc error: code = NotFound desc = could not find container \"827f8f579ad97a31a23a6e223c696bf65f534a64bf1811da87bec5389984bd59\": container with ID starting with 827f8f579ad97a31a23a6e223c696bf65f534a64bf1811da87bec5389984bd59 not found: ID does not exist" Oct 30 00:29:33.429424 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:33.429390 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmrvj\" (UniqueName: \"kubernetes.io/projected/17582778-cc63-4b44-ac34-ee9a4c866bba-kube-api-access-jmrvj\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:33.489486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:33.489462 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422c9df5-v25fr" Oct 30 00:29:33.530060 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:33.530030 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9aa57dc-4f61-4001-8573-9cd8ee69246f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9aa57dc-4f61-4001-8573-9cd8ee69246f-maas-api-openshift-ca\") pod \"a9aa57dc-4f61-4001-8573-9cd8ee69246f\" (UID: \"a9aa57dc-4f61-4001-8573-9cd8ee69246f\") " Oct 30 00:29:33.530234 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:33.530144 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9aa57dc-4f61-4001-8573-9cd8ee69246f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9aa57dc-4f61-4001-8573-9cd8ee69246f-maas-api-xks-ca\") pod \"a9aa57dc-4f61-4001-8573-9cd8ee69246f\" (UID: \"a9aa57dc-4f61-4001-8573-9cd8ee69246f\") " Oct 30 00:29:33.530234 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:33.530167 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9aa57dc-4f61-4001-8573-9cd8ee69246f-kube-api-access-7kmw2\" (UniqueName: \"kubernetes.io/projected/a9aa57dc-4f61-4001-8573-9cd8ee69246f-kube-api-access-7kmw2\") pod \"a9aa57dc-4f61-4001-8573-9cd8ee69246f\" (UID: \"a9aa57dc-4f61-4001-8573-9cd8ee69246f\") " Oct 30 00:29:33.530436 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:33.530412 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9aa57dc-4f61-4001-8573-9cd8ee69246f-maas-api-openshift-ca" pod "a9aa57dc-4f61-4001-8573-9cd8ee69246f" (UID: "a9aa57dc-4f61-4001-8573-9cd8ee69246f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:29:33.530904 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:33.530875 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9aa57dc-4f61-4001-8573-9cd8ee69246f-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:33.532430 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:33.532405 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9aa57dc-4f61-4001-8573-9cd8ee69246f-kube-api-access-7kmw2" pod "a9aa57dc-4f61-4001-8573-9cd8ee69246f" (UID: "a9aa57dc-4f61-4001-8573-9cd8ee69246f"). InnerVolumeSpecName "kube-api-access-7kmw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:29:33.532863 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:33.532842 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9aa57dc-4f61-4001-8573-9cd8ee69246f-maas-api-xks-ca" pod "a9aa57dc-4f61-4001-8573-9cd8ee69246f" (UID: "a9aa57dc-4f61-4001-8573-9cd8ee69246f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:29:33.632875 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:33.632849 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9aa57dc-4f61-4001-8573-9cd8ee69246f-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:33.632989 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:33.632881 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kmw2\" (UniqueName: \"kubernetes.io/projected/a9aa57dc-4f61-4001-8573-9cd8ee69246f-kube-api-access-7kmw2\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:33.667669 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:33.667639 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-5658554c47-sq9k7"] Oct 30 00:29:33.672045 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:33.672019 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a15230-5658554c47-sq9k7"] Oct 30 00:29:34.336509 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:34.336479 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422c9df5-v25fr" Oct 30 00:29:34.336685 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:34.336560 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422c9df5-v25fr" event={"ID":"a9aa57dc-4f61-4001-8573-9cd8ee69246f","Type":"ContainerDied","Data":"8b5a1650b5bd9cca3c48241aa6e52998cb71c74befa8add88a1b93ac886542af"} Oct 30 00:29:34.336685 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:34.336620 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b5a1650b5bd9cca3c48241aa6e52998cb71c74befa8add88a1b93ac886542af" Oct 30 00:29:34.920545 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:34.920517 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17582778-cc63-4b44-ac34-ee9a4c866bba" path="/var/lib/kubelet/pods/17582778-cc63-4b44-ac34-ee9a4c866bba/volumes" Oct 30 00:29:34.921026 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:34.921009 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ae0b29f-6fd9-494f-b22c-e313c324615f" path="/var/lib/kubelet/pods/3ae0b29f-6fd9-494f-b22c-e313c324615f/volumes" Oct 30 00:29:36.146950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.111980 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3eaadd9-xq2m8"] Oct 30 00:29:36.146950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.113055 5001 cpu_manager.go:414] "containerMap: removing container" podUID="a684c9be-f1fe-420b-a3a0-16ec60772131" containerName="istio-proxy" Oct 30 00:29:36.146950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.113080 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a684c9be-f1fe-420b-a3a0-16ec60772131" containerName="istio-proxy" Oct 30 00:29:36.146950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.113105 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1b18b95b-c9bb-4019-a6fe-79435057127c" containerName="payload-processing" Oct 30 00:29:36.146950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.113114 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b18b95b-c9bb-4019-a6fe-79435057127c" containerName="payload-processing" Oct 30 00:29:36.146950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.113158 5001 cpu_manager.go:414] "containerMap: removing container" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" Oct 30 00:29:36.146950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.113165 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" Oct 30 00:29:36.146950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.113199 5001 cpu_manager.go:414] "containerMap: removing container" podUID="a9aa57dc-4f61-4001-8573-9cd8ee69246f" containerName="revoke-keys" Oct 30 00:29:36.146950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.113205 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9aa57dc-4f61-4001-8573-9cd8ee69246f" containerName="revoke-keys" Oct 30 00:29:36.146950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.113220 5001 cpu_manager.go:414] "containerMap: removing container" podUID="17582778-cc63-4b44-ac34-ee9a4c866bba" containerName="payload-pre-processing" Oct 30 00:29:36.146950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.113228 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17582778-cc63-4b44-ac34-ee9a4c866bba" containerName="payload-pre-processing" Oct 30 00:29:36.146950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.113247 5001 cpu_manager.go:414] "containerMap: removing container" podUID="a684c9be-f1fe-420b-a3a0-16ec60772131" containerName="istio-proxy" Oct 30 00:29:36.146950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.113253 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a684c9be-f1fe-420b-a3a0-16ec60772131" containerName="istio-proxy" Oct 30 00:29:36.146950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.113423 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b18b95b-c9bb-4019-a6fe-79435057127c" containerName="payload-processing" Oct 30 00:29:36.146950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.113447 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c35f644-80ac-439f-9efd-631b26b20a0c" containerName="istio-proxy" Oct 30 00:29:36.146950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.113461 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9aa57dc-4f61-4001-8573-9cd8ee69246f" containerName="revoke-keys" Oct 30 00:29:36.146950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.113471 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="17582778-cc63-4b44-ac34-ee9a4c866bba" containerName="payload-pre-processing" Oct 30 00:29:36.146950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.113482 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a684c9be-f1fe-420b-a3a0-16ec60772131" containerName="istio-proxy" Oct 30 00:29:36.146950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.113494 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a684c9be-f1fe-420b-a3a0-16ec60772131" containerName="istio-proxy" Oct 30 00:29:36.146950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.144243 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3eaadd9-xq2m8" Oct 30 00:29:36.193530 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.166316 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:29:36.193530 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.179314 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:29:36.193530 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.179431 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:29:36.193530 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.179525 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:29:36.217937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.199500 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/109a1b04-b1d5-4d4e-ae59-c459fb040a6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3eaadd9-xq2m8\" (UID: \"109a1b04-b1d5-4d4e-ae59-c459fb040a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3eaadd9-xq2m8" Oct 30 00:29:36.217937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.199648 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/109a1b04-b1d5-4d4e-ae59-c459fb040a6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3eaadd9-xq2m8\" (UID: \"109a1b04-b1d5-4d4e-ae59-c459fb040a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3eaadd9-xq2m8" Oct 30 00:29:36.217937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.199685 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcr2s\" (UniqueName: \"kubernetes.io/projected/109a1b04-b1d5-4d4e-ae59-c459fb040a6d-kube-api-access-lcr2s\") pod \"maas-api-revoke-sub-f3eaadd9-xq2m8\" (UID: \"109a1b04-b1d5-4d4e-ae59-c459fb040a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3eaadd9-xq2m8" Oct 30 00:29:36.251938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.250869 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3eaadd9-xq2m8"] Oct 30 00:29:36.301278 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.301248 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/109a1b04-b1d5-4d4e-ae59-c459fb040a6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3eaadd9-xq2m8\" (UID: \"109a1b04-b1d5-4d4e-ae59-c459fb040a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3eaadd9-xq2m8" Oct 30 00:29:36.301451 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.301285 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcr2s\" (UniqueName: \"kubernetes.io/projected/109a1b04-b1d5-4d4e-ae59-c459fb040a6d-kube-api-access-lcr2s\") pod \"maas-api-revoke-sub-f3eaadd9-xq2m8\" (UID: \"109a1b04-b1d5-4d4e-ae59-c459fb040a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3eaadd9-xq2m8" Oct 30 00:29:36.301507 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.301468 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/109a1b04-b1d5-4d4e-ae59-c459fb040a6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3eaadd9-xq2m8\" (UID: \"109a1b04-b1d5-4d4e-ae59-c459fb040a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3eaadd9-xq2m8" Oct 30 00:29:36.302000 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.301934 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/109a1b04-b1d5-4d4e-ae59-c459fb040a6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3eaadd9-xq2m8\" (UID: \"109a1b04-b1d5-4d4e-ae59-c459fb040a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3eaadd9-xq2m8" Oct 30 00:29:36.303968 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.303897 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/109a1b04-b1d5-4d4e-ae59-c459fb040a6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3eaadd9-xq2m8\" (UID: \"109a1b04-b1d5-4d4e-ae59-c459fb040a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3eaadd9-xq2m8" Oct 30 00:29:36.321733 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.321709 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcr2s\" (UniqueName: \"kubernetes.io/projected/109a1b04-b1d5-4d4e-ae59-c459fb040a6d-kube-api-access-lcr2s\") pod \"maas-api-revoke-sub-f3eaadd9-xq2m8\" (UID: \"109a1b04-b1d5-4d4e-ae59-c459fb040a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3eaadd9-xq2m8" Oct 30 00:29:36.514829 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.514792 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3eaadd9-xq2m8" Oct 30 00:29:36.826447 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:29:36.826409 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod109a1b04_b1d5_4d4e_ae59_c459fb040a6d.slice/crio-2398aa224aaf6b096f8c191a282f3be4dac818e6a979dbcd1935f8a079d125f2 WatchSource:0}: Error finding container 2398aa224aaf6b096f8c191a282f3be4dac818e6a979dbcd1935f8a079d125f2: Status 404 returned error can't find the container with id 2398aa224aaf6b096f8c191a282f3be4dac818e6a979dbcd1935f8a079d125f2 Oct 30 00:29:36.826733 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:36.826708 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3eaadd9-xq2m8"] Oct 30 00:29:37.368165 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:37.368131 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3eaadd9-xq2m8" event={"ID":"109a1b04-b1d5-4d4e-ae59-c459fb040a6d","Type":"ContainerStarted","Data":"639dfaddabb608fb8c564e1f006985ba8e03981b71ec847a9f5ea60f8f64161f"} Oct 30 00:29:37.368512 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:37.368185 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3eaadd9-xq2m8" event={"ID":"109a1b04-b1d5-4d4e-ae59-c459fb040a6d","Type":"ContainerStarted","Data":"2398aa224aaf6b096f8c191a282f3be4dac818e6a979dbcd1935f8a079d125f2"} Oct 30 00:29:37.434109 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:37.426353 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3eaadd9-xq2m8" podStartSLOduration=1.42632884 podStartE2EDuration="1.42632884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:29:37.424161497 +0000 UTC m=+1492.629662680" watchObservedRunningTime="2026-10-30 00:29:37.42632884 +0000 UTC m=+1492.631830022" Oct 30 00:29:37.678775 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:37.678749 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:29:37.831152 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:37.831123 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/242dd865-71a7-4ade-a9ae-e80adc01cf59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/242dd865-71a7-4ade-a9ae-e80adc01cf59-istiod-ca-cert\") pod \"242dd865-71a7-4ade-a9ae-e80adc01cf59\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " Oct 30 00:29:37.831314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:37.831276 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-workload-socket\") pod \"242dd865-71a7-4ade-a9ae-e80adc01cf59\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " Oct 30 00:29:37.831373 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:37.831330 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-istio-data\") pod \"242dd865-71a7-4ade-a9ae-e80adc01cf59\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " Oct 30 00:29:37.831436 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:37.831390 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-workload-certs\") pod \"242dd865-71a7-4ade-a9ae-e80adc01cf59\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " Oct 30 00:29:37.831529 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:37.831513 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-credential-socket\") pod \"242dd865-71a7-4ade-a9ae-e80adc01cf59\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " Oct 30 00:29:37.831597 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:37.831529 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/242dd865-71a7-4ade-a9ae-e80adc01cf59-istiod-ca-cert" pod "242dd865-71a7-4ade-a9ae-e80adc01cf59" (UID: "242dd865-71a7-4ade-a9ae-e80adc01cf59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:29:37.831597 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:37.831568 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/242dd865-71a7-4ade-a9ae-e80adc01cf59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/242dd865-71a7-4ade-a9ae-e80adc01cf59-istio-podinfo\") pod \"242dd865-71a7-4ade-a9ae-e80adc01cf59\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " Oct 30 00:29:37.831694 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:37.831598 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-workload-socket" pod "242dd865-71a7-4ade-a9ae-e80adc01cf59" (UID: "242dd865-71a7-4ade-a9ae-e80adc01cf59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:29:37.831694 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:37.831624 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/242dd865-71a7-4ade-a9ae-e80adc01cf59-kube-api-access-8j669\" (UniqueName: \"kubernetes.io/projected/242dd865-71a7-4ade-a9ae-e80adc01cf59-kube-api-access-8j669\") pod \"242dd865-71a7-4ade-a9ae-e80adc01cf59\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " Oct 30 00:29:37.831694 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:37.831660 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-istio-data" pod "242dd865-71a7-4ade-a9ae-e80adc01cf59" (UID: "242dd865-71a7-4ade-a9ae-e80adc01cf59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:29:37.831694 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:37.831681 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/242dd865-71a7-4ade-a9ae-e80adc01cf59-istio-token\" (UniqueName: \"kubernetes.io/projected/242dd865-71a7-4ade-a9ae-e80adc01cf59-istio-token\") pod \"242dd865-71a7-4ade-a9ae-e80adc01cf59\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " Oct 30 00:29:37.831924 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:37.831701 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-workload-certs" pod "242dd865-71a7-4ade-a9ae-e80adc01cf59" (UID: "242dd865-71a7-4ade-a9ae-e80adc01cf59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:29:37.831924 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:37.831721 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-istio-envoy\") pod \"242dd865-71a7-4ade-a9ae-e80adc01cf59\" (UID: \"242dd865-71a7-4ade-a9ae-e80adc01cf59\") " Oct 30 00:29:37.831924 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:37.831892 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-credential-socket" pod "242dd865-71a7-4ade-a9ae-e80adc01cf59" (UID: "242dd865-71a7-4ade-a9ae-e80adc01cf59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:29:37.832476 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:37.832451 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:37.832616 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:37.832479 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:37.832616 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:37.832496 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/242dd865-71a7-4ade-a9ae-e80adc01cf59-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:37.832616 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:37.832509 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:37.832616 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:37.832524 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:37.834353 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:37.834333 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-istio-envoy" pod "242dd865-71a7-4ade-a9ae-e80adc01cf59" (UID: "242dd865-71a7-4ade-a9ae-e80adc01cf59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:29:37.834353 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:37.834339 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/242dd865-71a7-4ade-a9ae-e80adc01cf59-istio-podinfo" pod "242dd865-71a7-4ade-a9ae-e80adc01cf59" (UID: "242dd865-71a7-4ade-a9ae-e80adc01cf59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:29:37.834554 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:37.834348 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/242dd865-71a7-4ade-a9ae-e80adc01cf59-kube-api-access-8j669" pod "242dd865-71a7-4ade-a9ae-e80adc01cf59" (UID: "242dd865-71a7-4ade-a9ae-e80adc01cf59"). InnerVolumeSpecName "kube-api-access-8j669". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:29:37.834554 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:37.834383 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/242dd865-71a7-4ade-a9ae-e80adc01cf59-istio-token" pod "242dd865-71a7-4ade-a9ae-e80adc01cf59" (UID: "242dd865-71a7-4ade-a9ae-e80adc01cf59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:29:37.934503 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:37.934457 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/242dd865-71a7-4ade-a9ae-e80adc01cf59-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:37.934503 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:37.934490 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j669\" (UniqueName: \"kubernetes.io/projected/242dd865-71a7-4ade-a9ae-e80adc01cf59-kube-api-access-8j669\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:37.934503 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:37.934510 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/242dd865-71a7-4ade-a9ae-e80adc01cf59-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:37.934743 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:37.934519 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/242dd865-71a7-4ade-a9ae-e80adc01cf59-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:38.381706 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:38.381671 5001 generic.go:356] "Generic (PLEG): container finished" podID="109a1b04-b1d5-4d4e-ae59-c459fb040a6d" containerID="639dfaddabb608fb8c564e1f006985ba8e03981b71ec847a9f5ea60f8f64161f" exitCode=0 Oct 30 00:29:38.382204 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:38.381753 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3eaadd9-xq2m8" event={"ID":"109a1b04-b1d5-4d4e-ae59-c459fb040a6d","Type":"ContainerDied","Data":"639dfaddabb608fb8c564e1f006985ba8e03981b71ec847a9f5ea60f8f64161f"} Oct 30 00:29:38.384019 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:38.383998 5001 generic.go:356] "Generic (PLEG): container finished" podID="242dd865-71a7-4ade-a9ae-e80adc01cf59" containerID="5685b64fd757df5c4a8346e65c88697a2632d6a052a4faf76e483b3e0ed92539" exitCode=0 Oct 30 00:29:38.384141 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:38.384070 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" event={"ID":"242dd865-71a7-4ade-a9ae-e80adc01cf59","Type":"ContainerDied","Data":"5685b64fd757df5c4a8346e65c88697a2632d6a052a4faf76e483b3e0ed92539"} Oct 30 00:29:38.384141 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:38.384075 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" Oct 30 00:29:38.384141 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:38.384104 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd" event={"ID":"242dd865-71a7-4ade-a9ae-e80adc01cf59","Type":"ContainerDied","Data":"e02f88653545461aae9d2445c0adf04fe876b9c88547a064b7152c551351fa69"} Oct 30 00:29:38.384141 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:38.384131 5001 scope.go:122] "RemoveContainer" containerID="5685b64fd757df5c4a8346e65c88697a2632d6a052a4faf76e483b3e0ed92539" Oct 30 00:29:38.397873 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:38.397844 5001 scope.go:122] "RemoveContainer" containerID="3ca7cb7aa17eff4d5d03a35e8762ef1fc12a47cab61561c39e84758377dee015" Oct 30 00:29:38.408577 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:38.408559 5001 scope.go:122] "RemoveContainer" containerID="5685b64fd757df5c4a8346e65c88697a2632d6a052a4faf76e483b3e0ed92539" Oct 30 00:29:38.408862 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:29:38.408841 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5685b64fd757df5c4a8346e65c88697a2632d6a052a4faf76e483b3e0ed92539\": container with ID starting with 5685b64fd757df5c4a8346e65c88697a2632d6a052a4faf76e483b3e0ed92539 not found: ID does not exist" containerID="5685b64fd757df5c4a8346e65c88697a2632d6a052a4faf76e483b3e0ed92539" Oct 30 00:29:38.408990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:38.408867 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5685b64fd757df5c4a8346e65c88697a2632d6a052a4faf76e483b3e0ed92539"} err="failed to get container status \"5685b64fd757df5c4a8346e65c88697a2632d6a052a4faf76e483b3e0ed92539\": rpc error: code = NotFound desc = could not find container \"5685b64fd757df5c4a8346e65c88697a2632d6a052a4faf76e483b3e0ed92539\": container with ID starting with 5685b64fd757df5c4a8346e65c88697a2632d6a052a4faf76e483b3e0ed92539 not found: ID does not exist" Oct 30 00:29:38.408990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:38.408885 5001 scope.go:122] "RemoveContainer" containerID="3ca7cb7aa17eff4d5d03a35e8762ef1fc12a47cab61561c39e84758377dee015" Oct 30 00:29:38.409136 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:29:38.409123 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ca7cb7aa17eff4d5d03a35e8762ef1fc12a47cab61561c39e84758377dee015\": container with ID starting with 3ca7cb7aa17eff4d5d03a35e8762ef1fc12a47cab61561c39e84758377dee015 not found: ID does not exist" containerID="3ca7cb7aa17eff4d5d03a35e8762ef1fc12a47cab61561c39e84758377dee015" Oct 30 00:29:38.409182 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:38.409140 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ca7cb7aa17eff4d5d03a35e8762ef1fc12a47cab61561c39e84758377dee015"} err="failed to get container status \"3ca7cb7aa17eff4d5d03a35e8762ef1fc12a47cab61561c39e84758377dee015\": rpc error: code = NotFound desc = could not find container \"3ca7cb7aa17eff4d5d03a35e8762ef1fc12a47cab61561c39e84758377dee015\": container with ID starting with 3ca7cb7aa17eff4d5d03a35e8762ef1fc12a47cab61561c39e84758377dee015 not found: ID does not exist" Oct 30 00:29:38.451823 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:38.451787 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd"] Oct 30 00:29:38.459861 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:38.459835 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-4500d5-openshift-default-7f49c9dcc-dvbdd"] Oct 30 00:29:38.921853 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:38.921822 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="242dd865-71a7-4ade-a9ae-e80adc01cf59" path="/var/lib/kubelet/pods/242dd865-71a7-4ade-a9ae-e80adc01cf59/volumes" Oct 30 00:29:39.558987 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:39.558960 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3eaadd9-xq2m8" Oct 30 00:29:39.665771 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:39.665685 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/109a1b04-b1d5-4d4e-ae59-c459fb040a6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/109a1b04-b1d5-4d4e-ae59-c459fb040a6d-maas-api-openshift-ca\") pod \"109a1b04-b1d5-4d4e-ae59-c459fb040a6d\" (UID: \"109a1b04-b1d5-4d4e-ae59-c459fb040a6d\") " Oct 30 00:29:39.665974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:39.665785 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/109a1b04-b1d5-4d4e-ae59-c459fb040a6d-kube-api-access-lcr2s\" (UniqueName: \"kubernetes.io/projected/109a1b04-b1d5-4d4e-ae59-c459fb040a6d-kube-api-access-lcr2s\") pod \"109a1b04-b1d5-4d4e-ae59-c459fb040a6d\" (UID: \"109a1b04-b1d5-4d4e-ae59-c459fb040a6d\") " Oct 30 00:29:39.665974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:39.665816 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/109a1b04-b1d5-4d4e-ae59-c459fb040a6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/109a1b04-b1d5-4d4e-ae59-c459fb040a6d-maas-api-xks-ca\") pod \"109a1b04-b1d5-4d4e-ae59-c459fb040a6d\" (UID: \"109a1b04-b1d5-4d4e-ae59-c459fb040a6d\") " Oct 30 00:29:39.666166 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:39.666142 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/109a1b04-b1d5-4d4e-ae59-c459fb040a6d-maas-api-openshift-ca" pod "109a1b04-b1d5-4d4e-ae59-c459fb040a6d" (UID: "109a1b04-b1d5-4d4e-ae59-c459fb040a6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:29:39.666371 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:39.666347 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/109a1b04-b1d5-4d4e-ae59-c459fb040a6d-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:39.668052 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:39.668032 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/109a1b04-b1d5-4d4e-ae59-c459fb040a6d-kube-api-access-lcr2s" pod "109a1b04-b1d5-4d4e-ae59-c459fb040a6d" (UID: "109a1b04-b1d5-4d4e-ae59-c459fb040a6d"). InnerVolumeSpecName "kube-api-access-lcr2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:29:39.668347 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:39.668317 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/109a1b04-b1d5-4d4e-ae59-c459fb040a6d-maas-api-xks-ca" pod "109a1b04-b1d5-4d4e-ae59-c459fb040a6d" (UID: "109a1b04-b1d5-4d4e-ae59-c459fb040a6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:29:39.768096 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:39.768067 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcr2s\" (UniqueName: \"kubernetes.io/projected/109a1b04-b1d5-4d4e-ae59-c459fb040a6d-kube-api-access-lcr2s\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:39.768096 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:39.768092 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/109a1b04-b1d5-4d4e-ae59-c459fb040a6d-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:40.139071 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:40.139038 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw"] Oct 30 00:29:40.139540 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:40.139510 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" containerID="cri-o://6bdbe2e5223a5aa49ca43894b9b8c419233841bc1b50dad4504dbf4c16c696ff" gracePeriod=30 Oct 30 00:29:40.616731 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:40.615094 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3eaadd9-xq2m8" event={"ID":"109a1b04-b1d5-4d4e-ae59-c459fb040a6d","Type":"ContainerDied","Data":"2398aa224aaf6b096f8c191a282f3be4dac818e6a979dbcd1935f8a079d125f2"} Oct 30 00:29:40.616731 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:40.615179 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2398aa224aaf6b096f8c191a282f3be4dac818e6a979dbcd1935f8a079d125f2" Oct 30 00:29:40.671941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:40.664057 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3eaadd9-xq2m8" Oct 30 00:29:40.678837 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:40.676227 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-20d942-5bgmc"] Oct 30 00:29:40.682747 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:40.682721 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-20d942-5bgmc"] Oct 30 00:29:40.920340 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:40.920264 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b87b7cf9-722c-471c-8bc0-ffc576905258" path="/var/lib/kubelet/pods/b87b7cf9-722c-471c-8bc0-ffc576905258/volumes" Oct 30 00:29:42.737948 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:42.737891 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e4a714d-qwcz9"] Oct 30 00:29:42.738579 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:42.738563 5001 cpu_manager.go:414] "containerMap: removing container" podUID="109a1b04-b1d5-4d4e-ae59-c459fb040a6d" containerName="revoke-keys" Oct 30 00:29:42.738641 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:42.738580 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="109a1b04-b1d5-4d4e-ae59-c459fb040a6d" containerName="revoke-keys" Oct 30 00:29:42.738641 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:42.738601 5001 cpu_manager.go:414] "containerMap: removing container" podUID="242dd865-71a7-4ade-a9ae-e80adc01cf59" containerName="istio-proxy" Oct 30 00:29:42.738641 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:42.738606 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="242dd865-71a7-4ade-a9ae-e80adc01cf59" containerName="istio-proxy" Oct 30 00:29:42.738641 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:42.738615 5001 cpu_manager.go:414] "containerMap: removing container" podUID="242dd865-71a7-4ade-a9ae-e80adc01cf59" containerName="istio-proxy" Oct 30 00:29:42.738641 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:42.738620 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="242dd865-71a7-4ade-a9ae-e80adc01cf59" containerName="istio-proxy" Oct 30 00:29:42.738792 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:42.738742 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="109a1b04-b1d5-4d4e-ae59-c459fb040a6d" containerName="revoke-keys" Oct 30 00:29:42.738792 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:42.738753 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="242dd865-71a7-4ade-a9ae-e80adc01cf59" containerName="istio-proxy" Oct 30 00:29:42.738792 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:42.738760 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="242dd865-71a7-4ade-a9ae-e80adc01cf59" containerName="istio-proxy" Oct 30 00:29:42.743793 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:42.743772 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a714d-qwcz9" Oct 30 00:29:42.757472 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:42.757440 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:29:42.757472 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:42.757446 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:29:42.767351 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:42.767326 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:29:42.767351 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:42.767332 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:29:42.772292 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:42.772268 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e4a714d-qwcz9"] Oct 30 00:29:42.796362 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:42.796331 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f02d5c98-9a5f-4e40-b918-4c154e1e1c6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e4a714d-qwcz9\" (UID: \"f02d5c98-9a5f-4e40-b918-4c154e1e1c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a714d-qwcz9" Oct 30 00:29:42.796573 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:42.796549 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cszrf\" (UniqueName: \"kubernetes.io/projected/f02d5c98-9a5f-4e40-b918-4c154e1e1c6b-kube-api-access-cszrf\") pod \"maas-api-revoke-sub-5e4a714d-qwcz9\" (UID: \"f02d5c98-9a5f-4e40-b918-4c154e1e1c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a714d-qwcz9" Oct 30 00:29:42.796707 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:42.796684 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f02d5c98-9a5f-4e40-b918-4c154e1e1c6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e4a714d-qwcz9\" (UID: \"f02d5c98-9a5f-4e40-b918-4c154e1e1c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a714d-qwcz9" Oct 30 00:29:42.898557 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:42.898514 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cszrf\" (UniqueName: \"kubernetes.io/projected/f02d5c98-9a5f-4e40-b918-4c154e1e1c6b-kube-api-access-cszrf\") pod \"maas-api-revoke-sub-5e4a714d-qwcz9\" (UID: \"f02d5c98-9a5f-4e40-b918-4c154e1e1c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a714d-qwcz9" Oct 30 00:29:42.898736 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:42.898639 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f02d5c98-9a5f-4e40-b918-4c154e1e1c6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e4a714d-qwcz9\" (UID: \"f02d5c98-9a5f-4e40-b918-4c154e1e1c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a714d-qwcz9" Oct 30 00:29:42.898777 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:42.898753 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f02d5c98-9a5f-4e40-b918-4c154e1e1c6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e4a714d-qwcz9\" (UID: \"f02d5c98-9a5f-4e40-b918-4c154e1e1c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a714d-qwcz9" Oct 30 00:29:42.899344 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:42.899324 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f02d5c98-9a5f-4e40-b918-4c154e1e1c6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e4a714d-qwcz9\" (UID: \"f02d5c98-9a5f-4e40-b918-4c154e1e1c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a714d-qwcz9" Oct 30 00:29:42.901106 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:42.901085 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f02d5c98-9a5f-4e40-b918-4c154e1e1c6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e4a714d-qwcz9\" (UID: \"f02d5c98-9a5f-4e40-b918-4c154e1e1c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a714d-qwcz9" Oct 30 00:29:42.919540 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:42.919516 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cszrf\" (UniqueName: \"kubernetes.io/projected/f02d5c98-9a5f-4e40-b918-4c154e1e1c6b-kube-api-access-cszrf\") pod \"maas-api-revoke-sub-5e4a714d-qwcz9\" (UID: \"f02d5c98-9a5f-4e40-b918-4c154e1e1c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a714d-qwcz9" Oct 30 00:29:43.057772 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:43.057739 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a714d-qwcz9" Oct 30 00:29:43.146565 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:43.145366 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-520095f9-xm2fs"] Oct 30 00:29:43.150663 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:43.150641 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-520095f9-xm2fs" Oct 30 00:29:43.170145 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:43.170123 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-520095f9-xm2fs"] Oct 30 00:29:43.202718 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:43.202684 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls99h\" (UniqueName: \"kubernetes.io/projected/9559756a-57db-4a88-9a66-2889b5ead4af-kube-api-access-ls99h\") pod \"maas-api-revoke-sub-520095f9-xm2fs\" (UID: \"9559756a-57db-4a88-9a66-2889b5ead4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-520095f9-xm2fs" Oct 30 00:29:43.202897 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:43.202873 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9559756a-57db-4a88-9a66-2889b5ead4af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-520095f9-xm2fs\" (UID: \"9559756a-57db-4a88-9a66-2889b5ead4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-520095f9-xm2fs" Oct 30 00:29:43.202985 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:43.202941 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9559756a-57db-4a88-9a66-2889b5ead4af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-520095f9-xm2fs\" (UID: \"9559756a-57db-4a88-9a66-2889b5ead4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-520095f9-xm2fs" Oct 30 00:29:43.304747 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:43.304710 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls99h\" (UniqueName: \"kubernetes.io/projected/9559756a-57db-4a88-9a66-2889b5ead4af-kube-api-access-ls99h\") pod \"maas-api-revoke-sub-520095f9-xm2fs\" (UID: \"9559756a-57db-4a88-9a66-2889b5ead4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-520095f9-xm2fs" Oct 30 00:29:43.304977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:43.304824 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9559756a-57db-4a88-9a66-2889b5ead4af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-520095f9-xm2fs\" (UID: \"9559756a-57db-4a88-9a66-2889b5ead4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-520095f9-xm2fs" Oct 30 00:29:43.304977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:43.304862 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9559756a-57db-4a88-9a66-2889b5ead4af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-520095f9-xm2fs\" (UID: \"9559756a-57db-4a88-9a66-2889b5ead4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-520095f9-xm2fs" Oct 30 00:29:43.305587 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:43.305562 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9559756a-57db-4a88-9a66-2889b5ead4af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-520095f9-xm2fs\" (UID: \"9559756a-57db-4a88-9a66-2889b5ead4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-520095f9-xm2fs" Oct 30 00:29:43.307182 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:43.307161 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9559756a-57db-4a88-9a66-2889b5ead4af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-520095f9-xm2fs\" (UID: \"9559756a-57db-4a88-9a66-2889b5ead4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-520095f9-xm2fs" Oct 30 00:29:43.324994 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:43.324936 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls99h\" (UniqueName: \"kubernetes.io/projected/9559756a-57db-4a88-9a66-2889b5ead4af-kube-api-access-ls99h\") pod \"maas-api-revoke-sub-520095f9-xm2fs\" (UID: \"9559756a-57db-4a88-9a66-2889b5ead4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-520095f9-xm2fs" Oct 30 00:29:43.363963 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:29:43.363933 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf02d5c98_9a5f_4e40_b918_4c154e1e1c6b.slice/crio-1927bb3362fb9dd055e13a6c1c344aca67f9b04450796f203af7fb000ae02b82 WatchSource:0}: Error finding container 1927bb3362fb9dd055e13a6c1c344aca67f9b04450796f203af7fb000ae02b82: Status 404 returned error can't find the container with id 1927bb3362fb9dd055e13a6c1c344aca67f9b04450796f203af7fb000ae02b82 Oct 30 00:29:43.364512 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:43.364491 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e4a714d-qwcz9"] Oct 30 00:29:43.474812 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:43.474774 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-520095f9-xm2fs" Oct 30 00:29:43.757425 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:43.757386 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a714d-qwcz9" event={"ID":"f02d5c98-9a5f-4e40-b918-4c154e1e1c6b","Type":"ContainerStarted","Data":"1927bb3362fb9dd055e13a6c1c344aca67f9b04450796f203af7fb000ae02b82"} Oct 30 00:29:43.800461 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:43.800411 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-520095f9-xm2fs"] Oct 30 00:29:43.803049 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:29:43.803025 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9559756a_57db_4a88_9a66_2889b5ead4af.slice/crio-7302c5fb5e1f986cbcafa40075c37099f48c92f97b5458389c717878c97e7e63 WatchSource:0}: Error finding container 7302c5fb5e1f986cbcafa40075c37099f48c92f97b5458389c717878c97e7e63: Status 404 returned error can't find the container with id 7302c5fb5e1f986cbcafa40075c37099f48c92f97b5458389c717878c97e7e63 Oct 30 00:29:44.767979 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:44.767906 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-520095f9-xm2fs" event={"ID":"9559756a-57db-4a88-9a66-2889b5ead4af","Type":"ContainerStarted","Data":"ba63a7f2ece412f209dd5010fa6ae6e60e02f6370967b3dd8850665cc69ca53d"} Oct 30 00:29:44.768377 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:44.767997 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-520095f9-xm2fs" event={"ID":"9559756a-57db-4a88-9a66-2889b5ead4af","Type":"ContainerStarted","Data":"7302c5fb5e1f986cbcafa40075c37099f48c92f97b5458389c717878c97e7e63"} Oct 30 00:29:44.769670 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:44.769645 5001 generic.go:356] "Generic (PLEG): container finished" podID="f02d5c98-9a5f-4e40-b918-4c154e1e1c6b" containerID="fcbcc2caa4003d7edff0d1e3e44f8669d2603c74ddfb980f60d12efe776142b5" exitCode=0 Oct 30 00:29:44.769783 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:44.769695 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a714d-qwcz9" event={"ID":"f02d5c98-9a5f-4e40-b918-4c154e1e1c6b","Type":"ContainerDied","Data":"fcbcc2caa4003d7edff0d1e3e44f8669d2603c74ddfb980f60d12efe776142b5"} Oct 30 00:29:44.804596 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:44.804535 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-520095f9-xm2fs" podStartSLOduration=1.804518727 podStartE2EDuration="1.804518727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:29:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:29:44.802170881 +0000 UTC m=+1500.007672085" watchObservedRunningTime="2026-10-30 00:29:44.804518727 +0000 UTC m=+1500.010019907" Oct 30 00:29:45.301392 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.301358 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8gb5_9fa688d8-29ca-4c1f-926a-f1a4099e6b44/console-operator/2.log" Oct 30 00:29:45.308829 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.308750 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8gb5_9fa688d8-29ca-4c1f-926a-f1a4099e6b44/console-operator/2.log" Oct 30 00:29:45.451305 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.451278 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:29:45.548863 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.548827 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/809701ad-afba-418f-8844-36e0baa399df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/809701ad-afba-418f-8844-36e0baa399df-istio-podinfo\") pod \"809701ad-afba-418f-8844-36e0baa399df\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " Oct 30 00:29:45.549053 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.548966 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-istio-envoy\") pod \"809701ad-afba-418f-8844-36e0baa399df\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " Oct 30 00:29:45.549053 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.549039 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-workload-socket\") pod \"809701ad-afba-418f-8844-36e0baa399df\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " Oct 30 00:29:45.549217 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.549150 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-workload-certs\") pod \"809701ad-afba-418f-8844-36e0baa399df\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " Oct 30 00:29:45.549376 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.549335 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/809701ad-afba-418f-8844-36e0baa399df-kube-api-access-tbwwn\" (UniqueName: \"kubernetes.io/projected/809701ad-afba-418f-8844-36e0baa399df-kube-api-access-tbwwn\") pod \"809701ad-afba-418f-8844-36e0baa399df\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " Oct 30 00:29:45.549473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.549374 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-credential-socket\") pod \"809701ad-afba-418f-8844-36e0baa399df\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " Oct 30 00:29:45.549473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.549378 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-workload-socket" pod "809701ad-afba-418f-8844-36e0baa399df" (UID: "809701ad-afba-418f-8844-36e0baa399df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:29:45.549473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.549411 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-workload-certs" pod "809701ad-afba-418f-8844-36e0baa399df" (UID: "809701ad-afba-418f-8844-36e0baa399df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:29:45.549473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.549466 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-istio-data\") pod \"809701ad-afba-418f-8844-36e0baa399df\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " Oct 30 00:29:45.549708 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.549529 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/809701ad-afba-418f-8844-36e0baa399df-istio-token\" (UniqueName: \"kubernetes.io/projected/809701ad-afba-418f-8844-36e0baa399df-istio-token\") pod \"809701ad-afba-418f-8844-36e0baa399df\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " Oct 30 00:29:45.549708 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.549598 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/809701ad-afba-418f-8844-36e0baa399df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/809701ad-afba-418f-8844-36e0baa399df-istiod-ca-cert\") pod \"809701ad-afba-418f-8844-36e0baa399df\" (UID: \"809701ad-afba-418f-8844-36e0baa399df\") " Oct 30 00:29:45.549708 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.549664 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-credential-socket" pod "809701ad-afba-418f-8844-36e0baa399df" (UID: "809701ad-afba-418f-8844-36e0baa399df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:29:45.549892 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.549783 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-istio-data" pod "809701ad-afba-418f-8844-36e0baa399df" (UID: "809701ad-afba-418f-8844-36e0baa399df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:29:45.550298 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.550276 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:45.550413 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.550337 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:45.550413 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.550355 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:45.550413 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.550284 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/809701ad-afba-418f-8844-36e0baa399df-istiod-ca-cert" pod "809701ad-afba-418f-8844-36e0baa399df" (UID: "809701ad-afba-418f-8844-36e0baa399df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:29:45.550413 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.550370 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:45.551892 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.551854 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-istio-envoy" pod "809701ad-afba-418f-8844-36e0baa399df" (UID: "809701ad-afba-418f-8844-36e0baa399df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:29:45.552067 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.552043 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/809701ad-afba-418f-8844-36e0baa399df-istio-token" pod "809701ad-afba-418f-8844-36e0baa399df" (UID: "809701ad-afba-418f-8844-36e0baa399df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:29:45.552179 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.552140 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/809701ad-afba-418f-8844-36e0baa399df-istio-podinfo" pod "809701ad-afba-418f-8844-36e0baa399df" (UID: "809701ad-afba-418f-8844-36e0baa399df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:29:45.553881 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.553847 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/809701ad-afba-418f-8844-36e0baa399df-kube-api-access-tbwwn" pod "809701ad-afba-418f-8844-36e0baa399df" (UID: "809701ad-afba-418f-8844-36e0baa399df"). InnerVolumeSpecName "kube-api-access-tbwwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:29:45.652347 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.652314 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/809701ad-afba-418f-8844-36e0baa399df-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:45.652347 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.652342 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/809701ad-afba-418f-8844-36e0baa399df-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:45.652347 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.652354 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/809701ad-afba-418f-8844-36e0baa399df-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:45.652582 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.652364 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/809701ad-afba-418f-8844-36e0baa399df-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:45.652582 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.652375 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbwwn\" (UniqueName: \"kubernetes.io/projected/809701ad-afba-418f-8844-36e0baa399df-kube-api-access-tbwwn\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:45.780799 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.780766 5001 generic.go:356] "Generic (PLEG): container finished" podID="9559756a-57db-4a88-9a66-2889b5ead4af" containerID="ba63a7f2ece412f209dd5010fa6ae6e60e02f6370967b3dd8850665cc69ca53d" exitCode=0 Oct 30 00:29:45.781307 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.780890 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-520095f9-xm2fs" event={"ID":"9559756a-57db-4a88-9a66-2889b5ead4af","Type":"ContainerDied","Data":"ba63a7f2ece412f209dd5010fa6ae6e60e02f6370967b3dd8850665cc69ca53d"} Oct 30 00:29:45.782945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.782922 5001 generic.go:356] "Generic (PLEG): container finished" podID="809701ad-afba-418f-8844-36e0baa399df" containerID="6bdbe2e5223a5aa49ca43894b9b8c419233841bc1b50dad4504dbf4c16c696ff" exitCode=0 Oct 30 00:29:45.783088 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.782956 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" Oct 30 00:29:45.783088 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.783003 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" event={"ID":"809701ad-afba-418f-8844-36e0baa399df","Type":"ContainerDied","Data":"6bdbe2e5223a5aa49ca43894b9b8c419233841bc1b50dad4504dbf4c16c696ff"} Oct 30 00:29:45.783088 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.783059 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw" event={"ID":"809701ad-afba-418f-8844-36e0baa399df","Type":"ContainerDied","Data":"6966b0a85c287169ee050052aea8f7d4b5d9d9d30a09a54a1bdb4f215d5c0bd1"} Oct 30 00:29:45.783263 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.783091 5001 scope.go:122] "RemoveContainer" containerID="6bdbe2e5223a5aa49ca43894b9b8c419233841bc1b50dad4504dbf4c16c696ff" Oct 30 00:29:45.801745 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.801723 5001 scope.go:122] "RemoveContainer" containerID="bc3f1408c65e20151b5bc9f87e6baa20216ca6c4a1e183ec92d2074897819519" Oct 30 00:29:45.814110 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.814086 5001 scope.go:122] "RemoveContainer" containerID="6bdbe2e5223a5aa49ca43894b9b8c419233841bc1b50dad4504dbf4c16c696ff" Oct 30 00:29:45.814469 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:29:45.814440 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bdbe2e5223a5aa49ca43894b9b8c419233841bc1b50dad4504dbf4c16c696ff\": container with ID starting with 6bdbe2e5223a5aa49ca43894b9b8c419233841bc1b50dad4504dbf4c16c696ff not found: ID does not exist" containerID="6bdbe2e5223a5aa49ca43894b9b8c419233841bc1b50dad4504dbf4c16c696ff" Oct 30 00:29:45.814584 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.814478 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bdbe2e5223a5aa49ca43894b9b8c419233841bc1b50dad4504dbf4c16c696ff"} err="failed to get container status \"6bdbe2e5223a5aa49ca43894b9b8c419233841bc1b50dad4504dbf4c16c696ff\": rpc error: code = NotFound desc = could not find container \"6bdbe2e5223a5aa49ca43894b9b8c419233841bc1b50dad4504dbf4c16c696ff\": container with ID starting with 6bdbe2e5223a5aa49ca43894b9b8c419233841bc1b50dad4504dbf4c16c696ff not found: ID does not exist" Oct 30 00:29:45.814584 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.814502 5001 scope.go:122] "RemoveContainer" containerID="bc3f1408c65e20151b5bc9f87e6baa20216ca6c4a1e183ec92d2074897819519" Oct 30 00:29:45.814811 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:29:45.814788 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc3f1408c65e20151b5bc9f87e6baa20216ca6c4a1e183ec92d2074897819519\": container with ID starting with bc3f1408c65e20151b5bc9f87e6baa20216ca6c4a1e183ec92d2074897819519 not found: ID does not exist" containerID="bc3f1408c65e20151b5bc9f87e6baa20216ca6c4a1e183ec92d2074897819519" Oct 30 00:29:45.814927 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.814818 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc3f1408c65e20151b5bc9f87e6baa20216ca6c4a1e183ec92d2074897819519"} err="failed to get container status \"bc3f1408c65e20151b5bc9f87e6baa20216ca6c4a1e183ec92d2074897819519\": rpc error: code = NotFound desc = could not find container \"bc3f1408c65e20151b5bc9f87e6baa20216ca6c4a1e183ec92d2074897819519\": container with ID starting with bc3f1408c65e20151b5bc9f87e6baa20216ca6c4a1e183ec92d2074897819519 not found: ID does not exist" Oct 30 00:29:45.854350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.854144 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw"] Oct 30 00:29:45.864688 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.864655 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a15230-openshift-default-6bbf5764b9-4pxqw"] Oct 30 00:29:45.961002 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:45.960974 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a714d-qwcz9" Oct 30 00:29:46.058487 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:46.058389 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f02d5c98-9a5f-4e40-b918-4c154e1e1c6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f02d5c98-9a5f-4e40-b918-4c154e1e1c6b-maas-api-openshift-ca\") pod \"f02d5c98-9a5f-4e40-b918-4c154e1e1c6b\" (UID: \"f02d5c98-9a5f-4e40-b918-4c154e1e1c6b\") " Oct 30 00:29:46.058487 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:46.058459 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f02d5c98-9a5f-4e40-b918-4c154e1e1c6b-kube-api-access-cszrf\" (UniqueName: \"kubernetes.io/projected/f02d5c98-9a5f-4e40-b918-4c154e1e1c6b-kube-api-access-cszrf\") pod \"f02d5c98-9a5f-4e40-b918-4c154e1e1c6b\" (UID: \"f02d5c98-9a5f-4e40-b918-4c154e1e1c6b\") " Oct 30 00:29:46.058750 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:46.058543 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f02d5c98-9a5f-4e40-b918-4c154e1e1c6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f02d5c98-9a5f-4e40-b918-4c154e1e1c6b-maas-api-xks-ca\") pod \"f02d5c98-9a5f-4e40-b918-4c154e1e1c6b\" (UID: \"f02d5c98-9a5f-4e40-b918-4c154e1e1c6b\") " Oct 30 00:29:46.058819 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:46.058777 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f02d5c98-9a5f-4e40-b918-4c154e1e1c6b-maas-api-openshift-ca" pod "f02d5c98-9a5f-4e40-b918-4c154e1e1c6b" (UID: "f02d5c98-9a5f-4e40-b918-4c154e1e1c6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:29:46.059328 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:46.059301 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f02d5c98-9a5f-4e40-b918-4c154e1e1c6b-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:46.061248 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:46.061221 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f02d5c98-9a5f-4e40-b918-4c154e1e1c6b-kube-api-access-cszrf" pod "f02d5c98-9a5f-4e40-b918-4c154e1e1c6b" (UID: "f02d5c98-9a5f-4e40-b918-4c154e1e1c6b"). InnerVolumeSpecName "kube-api-access-cszrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:29:46.061391 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:46.061273 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f02d5c98-9a5f-4e40-b918-4c154e1e1c6b-maas-api-xks-ca" pod "f02d5c98-9a5f-4e40-b918-4c154e1e1c6b" (UID: "f02d5c98-9a5f-4e40-b918-4c154e1e1c6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:29:46.160687 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:46.160653 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cszrf\" (UniqueName: \"kubernetes.io/projected/f02d5c98-9a5f-4e40-b918-4c154e1e1c6b-kube-api-access-cszrf\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:46.160687 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:46.160683 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f02d5c98-9a5f-4e40-b918-4c154e1e1c6b-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:46.792585 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:46.792555 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a714d-qwcz9" Oct 30 00:29:46.793044 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:46.792592 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4a714d-qwcz9" event={"ID":"f02d5c98-9a5f-4e40-b918-4c154e1e1c6b","Type":"ContainerDied","Data":"1927bb3362fb9dd055e13a6c1c344aca67f9b04450796f203af7fb000ae02b82"} Oct 30 00:29:46.793044 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:46.792638 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1927bb3362fb9dd055e13a6c1c344aca67f9b04450796f203af7fb000ae02b82" Oct 30 00:29:46.926196 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:46.926155 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="809701ad-afba-418f-8844-36e0baa399df" path="/var/lib/kubelet/pods/809701ad-afba-418f-8844-36e0baa399df/volumes" Oct 30 00:29:47.065013 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:47.063354 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-520095f9-xm2fs" Oct 30 00:29:47.088964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:47.088928 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9559756a-57db-4a88-9a66-2889b5ead4af-kube-api-access-ls99h\" (UniqueName: \"kubernetes.io/projected/9559756a-57db-4a88-9a66-2889b5ead4af-kube-api-access-ls99h\") pod \"9559756a-57db-4a88-9a66-2889b5ead4af\" (UID: \"9559756a-57db-4a88-9a66-2889b5ead4af\") " Oct 30 00:29:47.089152 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:47.089094 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9559756a-57db-4a88-9a66-2889b5ead4af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9559756a-57db-4a88-9a66-2889b5ead4af-maas-api-xks-ca\") pod \"9559756a-57db-4a88-9a66-2889b5ead4af\" (UID: \"9559756a-57db-4a88-9a66-2889b5ead4af\") " Oct 30 00:29:47.089152 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:47.089146 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9559756a-57db-4a88-9a66-2889b5ead4af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9559756a-57db-4a88-9a66-2889b5ead4af-maas-api-openshift-ca\") pod \"9559756a-57db-4a88-9a66-2889b5ead4af\" (UID: \"9559756a-57db-4a88-9a66-2889b5ead4af\") " Oct 30 00:29:47.089589 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:47.089562 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9559756a-57db-4a88-9a66-2889b5ead4af-maas-api-openshift-ca" pod "9559756a-57db-4a88-9a66-2889b5ead4af" (UID: "9559756a-57db-4a88-9a66-2889b5ead4af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:29:47.089748 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:47.089729 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9559756a-57db-4a88-9a66-2889b5ead4af-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:47.091200 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:47.091179 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9559756a-57db-4a88-9a66-2889b5ead4af-maas-api-xks-ca" pod "9559756a-57db-4a88-9a66-2889b5ead4af" (UID: "9559756a-57db-4a88-9a66-2889b5ead4af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:29:47.091200 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:47.091185 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9559756a-57db-4a88-9a66-2889b5ead4af-kube-api-access-ls99h" pod "9559756a-57db-4a88-9a66-2889b5ead4af" (UID: "9559756a-57db-4a88-9a66-2889b5ead4af"). InnerVolumeSpecName "kube-api-access-ls99h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:29:47.192002 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:47.191968 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls99h\" (UniqueName: \"kubernetes.io/projected/9559756a-57db-4a88-9a66-2889b5ead4af-kube-api-access-ls99h\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:47.192002 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:47.192003 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9559756a-57db-4a88-9a66-2889b5ead4af-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:47.388282 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:29:47.388196 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:29:47.567340 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:47.567308 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk"] Oct 30 00:29:47.567598 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:47.567572 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" podUID="91eb024d-d850-4682-9821-bcae4cb40b49" containerName="istio-proxy" containerID="cri-o://197e965431543abb5219476fdc6bc90d090107bf85c869b387b45d177bde1651" gracePeriod=30 Oct 30 00:29:47.867935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:47.864859 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-520095f9-xm2fs" event={"ID":"9559756a-57db-4a88-9a66-2889b5ead4af","Type":"ContainerDied","Data":"7302c5fb5e1f986cbcafa40075c37099f48c92f97b5458389c717878c97e7e63"} Oct 30 00:29:47.867935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:47.864933 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7302c5fb5e1f986cbcafa40075c37099f48c92f97b5458389c717878c97e7e63" Oct 30 00:29:47.867935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:47.864998 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-520095f9-xm2fs" Oct 30 00:29:53.028330 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.028294 5001 generic.go:356] "Generic (PLEG): container finished" podID="91eb024d-d850-4682-9821-bcae4cb40b49" containerID="197e965431543abb5219476fdc6bc90d090107bf85c869b387b45d177bde1651" exitCode=0 Oct 30 00:29:53.029247 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.029220 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" event={"ID":"91eb024d-d850-4682-9821-bcae4cb40b49","Type":"ContainerDied","Data":"197e965431543abb5219476fdc6bc90d090107bf85c869b387b45d177bde1651"} Oct 30 00:29:53.029449 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.029433 5001 scope.go:122] "RemoveContainer" containerID="8eacc60f7c5c21d0a438dc1a6137c99db7dddde626d7aa5a33f5807edbde3bfa" Oct 30 00:29:53.138782 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.138754 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:29:53.199236 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.199162 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91eb024d-d850-4682-9821-bcae4cb40b49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91eb024d-d850-4682-9821-bcae4cb40b49-istio-podinfo\") pod \"91eb024d-d850-4682-9821-bcae4cb40b49\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " Oct 30 00:29:53.199408 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.199263 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-workload-socket\") pod \"91eb024d-d850-4682-9821-bcae4cb40b49\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " Oct 30 00:29:53.199408 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.199338 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-istio-data\") pod \"91eb024d-d850-4682-9821-bcae4cb40b49\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " Oct 30 00:29:53.199408 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.199370 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-istio-envoy\") pod \"91eb024d-d850-4682-9821-bcae4cb40b49\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " Oct 30 00:29:53.199599 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.199428 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91eb024d-d850-4682-9821-bcae4cb40b49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91eb024d-d850-4682-9821-bcae4cb40b49-istiod-ca-cert\") pod \"91eb024d-d850-4682-9821-bcae4cb40b49\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " Oct 30 00:29:53.199599 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.199525 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-credential-socket\") pod \"91eb024d-d850-4682-9821-bcae4cb40b49\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " Oct 30 00:29:53.199686 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.199659 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91eb024d-d850-4682-9821-bcae4cb40b49-istio-token\" (UniqueName: \"kubernetes.io/projected/91eb024d-d850-4682-9821-bcae4cb40b49-istio-token\") pod \"91eb024d-d850-4682-9821-bcae4cb40b49\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " Oct 30 00:29:53.199733 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.199698 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-workload-certs\") pod \"91eb024d-d850-4682-9821-bcae4cb40b49\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " Oct 30 00:29:53.199806 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.199789 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91eb024d-d850-4682-9821-bcae4cb40b49-kube-api-access-7phqh\" (UniqueName: \"kubernetes.io/projected/91eb024d-d850-4682-9821-bcae4cb40b49-kube-api-access-7phqh\") pod \"91eb024d-d850-4682-9821-bcae4cb40b49\" (UID: \"91eb024d-d850-4682-9821-bcae4cb40b49\") " Oct 30 00:29:53.201140 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.201115 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-workload-socket" pod "91eb024d-d850-4682-9821-bcae4cb40b49" (UID: "91eb024d-d850-4682-9821-bcae4cb40b49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:29:53.201607 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.201586 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91eb024d-d850-4682-9821-bcae4cb40b49-istiod-ca-cert" pod "91eb024d-d850-4682-9821-bcae4cb40b49" (UID: "91eb024d-d850-4682-9821-bcae4cb40b49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:29:53.201833 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.201689 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-credential-socket" pod "91eb024d-d850-4682-9821-bcae4cb40b49" (UID: "91eb024d-d850-4682-9821-bcae4cb40b49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:29:53.201977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.201757 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-workload-certs" pod "91eb024d-d850-4682-9821-bcae4cb40b49" (UID: "91eb024d-d850-4682-9821-bcae4cb40b49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:29:53.202299 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.202273 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-istio-data" pod "91eb024d-d850-4682-9821-bcae4cb40b49" (UID: "91eb024d-d850-4682-9821-bcae4cb40b49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:29:53.203420 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.203391 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-istio-envoy" pod "91eb024d-d850-4682-9821-bcae4cb40b49" (UID: "91eb024d-d850-4682-9821-bcae4cb40b49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:29:53.203662 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.203641 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91eb024d-d850-4682-9821-bcae4cb40b49-kube-api-access-7phqh" pod "91eb024d-d850-4682-9821-bcae4cb40b49" (UID: "91eb024d-d850-4682-9821-bcae4cb40b49"). InnerVolumeSpecName "kube-api-access-7phqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:29:53.211055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.211032 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91eb024d-d850-4682-9821-bcae4cb40b49-istio-podinfo" pod "91eb024d-d850-4682-9821-bcae4cb40b49" (UID: "91eb024d-d850-4682-9821-bcae4cb40b49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:29:53.211149 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.211085 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91eb024d-d850-4682-9821-bcae4cb40b49-istio-token" pod "91eb024d-d850-4682-9821-bcae4cb40b49" (UID: "91eb024d-d850-4682-9821-bcae4cb40b49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:29:53.302137 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.302101 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91eb024d-d850-4682-9821-bcae4cb40b49-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:53.302137 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.302132 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:53.302137 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.302141 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7phqh\" (UniqueName: \"kubernetes.io/projected/91eb024d-d850-4682-9821-bcae4cb40b49-kube-api-access-7phqh\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:53.302385 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.302153 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91eb024d-d850-4682-9821-bcae4cb40b49-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:53.302385 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.302162 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:53.302385 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.302171 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:53.302385 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.302180 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:53.302385 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.302188 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91eb024d-d850-4682-9821-bcae4cb40b49-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:53.302385 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.302197 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91eb024d-d850-4682-9821-bcae4cb40b49-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:53.548276 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.548241 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3536f0d4-tz9pw"] Oct 30 00:29:53.548871 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.548857 5001 cpu_manager.go:414] "containerMap: removing container" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" Oct 30 00:29:53.548938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.548872 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" Oct 30 00:29:53.548938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.548888 5001 cpu_manager.go:414] "containerMap: removing container" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" Oct 30 00:29:53.548938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.548893 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" Oct 30 00:29:53.548938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.548901 5001 cpu_manager.go:414] "containerMap: removing container" podUID="91eb024d-d850-4682-9821-bcae4cb40b49" containerName="istio-proxy" Oct 30 00:29:53.548938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.548906 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91eb024d-d850-4682-9821-bcae4cb40b49" containerName="istio-proxy" Oct 30 00:29:53.549145 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.548943 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f02d5c98-9a5f-4e40-b918-4c154e1e1c6b" containerName="revoke-keys" Oct 30 00:29:53.549145 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.548949 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f02d5c98-9a5f-4e40-b918-4c154e1e1c6b" containerName="revoke-keys" Oct 30 00:29:53.549145 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.548959 5001 cpu_manager.go:414] "containerMap: removing container" podUID="91eb024d-d850-4682-9821-bcae4cb40b49" containerName="istio-proxy" Oct 30 00:29:53.549145 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.548964 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91eb024d-d850-4682-9821-bcae4cb40b49" containerName="istio-proxy" Oct 30 00:29:53.549145 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.548979 5001 cpu_manager.go:414] "containerMap: removing container" podUID="9559756a-57db-4a88-9a66-2889b5ead4af" containerName="revoke-keys" Oct 30 00:29:53.549145 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.548984 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9559756a-57db-4a88-9a66-2889b5ead4af" containerName="revoke-keys" Oct 30 00:29:53.549145 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.549076 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="91eb024d-d850-4682-9821-bcae4cb40b49" containerName="istio-proxy" Oct 30 00:29:53.549145 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.549086 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9559756a-57db-4a88-9a66-2889b5ead4af" containerName="revoke-keys" Oct 30 00:29:53.549145 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.549096 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" Oct 30 00:29:53.549145 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.549103 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" Oct 30 00:29:53.549145 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.549108 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="91eb024d-d850-4682-9821-bcae4cb40b49" containerName="istio-proxy" Oct 30 00:29:53.549145 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.549118 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f02d5c98-9a5f-4e40-b918-4c154e1e1c6b" containerName="revoke-keys" Oct 30 00:29:53.552712 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.552692 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3536f0d4-tz9pw" Oct 30 00:29:53.568054 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.568021 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:29:53.568054 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.568045 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:29:53.568054 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.568028 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:29:53.568321 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.568198 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:29:53.573904 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.573880 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3536f0d4-tz9pw"] Oct 30 00:29:53.606856 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.606822 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3536f0d4-tz9pw\" (UID: \"4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3536f0d4-tz9pw" Oct 30 00:29:53.606856 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.606859 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3536f0d4-tz9pw\" (UID: \"4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3536f0d4-tz9pw" Oct 30 00:29:53.607138 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.607030 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-858bh\" (UniqueName: \"kubernetes.io/projected/4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838-kube-api-access-858bh\") pod \"maas-api-revoke-sub-3536f0d4-tz9pw\" (UID: \"4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3536f0d4-tz9pw" Oct 30 00:29:53.708527 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.708494 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3536f0d4-tz9pw\" (UID: \"4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3536f0d4-tz9pw" Oct 30 00:29:53.708709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.708538 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3536f0d4-tz9pw\" (UID: \"4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3536f0d4-tz9pw" Oct 30 00:29:53.708709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.708629 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-858bh\" (UniqueName: \"kubernetes.io/projected/4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838-kube-api-access-858bh\") pod \"maas-api-revoke-sub-3536f0d4-tz9pw\" (UID: \"4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3536f0d4-tz9pw" Oct 30 00:29:53.709307 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.709283 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3536f0d4-tz9pw\" (UID: \"4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3536f0d4-tz9pw" Oct 30 00:29:53.711010 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.710994 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3536f0d4-tz9pw\" (UID: \"4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3536f0d4-tz9pw" Oct 30 00:29:53.729152 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.729126 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-858bh\" (UniqueName: \"kubernetes.io/projected/4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838-kube-api-access-858bh\") pod \"maas-api-revoke-sub-3536f0d4-tz9pw\" (UID: \"4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3536f0d4-tz9pw" Oct 30 00:29:53.867691 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:53.867596 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3536f0d4-tz9pw" Oct 30 00:29:54.042029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:54.041992 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" Oct 30 00:29:54.042596 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:54.042031 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk" event={"ID":"91eb024d-d850-4682-9821-bcae4cb40b49","Type":"ContainerDied","Data":"680e4b2678ce03bd561c30306a579164c8d54518a6c9479162613a80d25eba2b"} Oct 30 00:29:54.042596 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:54.042090 5001 scope.go:122] "RemoveContainer" containerID="197e965431543abb5219476fdc6bc90d090107bf85c869b387b45d177bde1651" Oct 30 00:29:54.088925 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:54.088884 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk"] Oct 30 00:29:54.098609 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:54.098573 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-20d942-openshift-default-6c49b5c945-gcbsk"] Oct 30 00:29:54.186433 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:54.186396 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3536f0d4-tz9pw"] Oct 30 00:29:54.194006 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:29:54.193977 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dc85ca4_ac2f_4fd2_96fe_ba547ae9c838.slice/crio-e3737404aa5562ee2bef73b9517080010368fa23db49e14be858768a3fb05c04 WatchSource:0}: Error finding container e3737404aa5562ee2bef73b9517080010368fa23db49e14be858768a3fb05c04: Status 404 returned error can't find the container with id e3737404aa5562ee2bef73b9517080010368fa23db49e14be858768a3fb05c04 Oct 30 00:29:54.508671 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:54.508584 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7116507d-hx69n"] Oct 30 00:29:54.509512 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:54.509489 5001 cpu_manager.go:414] "containerMap: removing container" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" Oct 30 00:29:54.509512 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:54.509510 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" Oct 30 00:29:54.509687 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:54.509639 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="809701ad-afba-418f-8844-36e0baa399df" containerName="istio-proxy" Oct 30 00:29:54.514985 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:54.514961 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116507d-hx69n" Oct 30 00:29:54.533723 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:54.533659 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7116507d-hx69n"] Oct 30 00:29:54.636754 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:54.636731 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b810194b-c717-4bf4-8558-07eab131bdc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7116507d-hx69n\" (UID: \"b810194b-c717-4bf4-8558-07eab131bdc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116507d-hx69n" Oct 30 00:29:54.636851 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:54.636776 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b810194b-c717-4bf4-8558-07eab131bdc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7116507d-hx69n\" (UID: \"b810194b-c717-4bf4-8558-07eab131bdc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116507d-hx69n" Oct 30 00:29:54.636948 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:54.636927 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4kdm\" (UniqueName: \"kubernetes.io/projected/b810194b-c717-4bf4-8558-07eab131bdc5-kube-api-access-h4kdm\") pod \"maas-api-revoke-sub-7116507d-hx69n\" (UID: \"b810194b-c717-4bf4-8558-07eab131bdc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116507d-hx69n" Oct 30 00:29:54.738392 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:54.738353 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b810194b-c717-4bf4-8558-07eab131bdc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7116507d-hx69n\" (UID: \"b810194b-c717-4bf4-8558-07eab131bdc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116507d-hx69n" Oct 30 00:29:54.738580 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:54.738420 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b810194b-c717-4bf4-8558-07eab131bdc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7116507d-hx69n\" (UID: \"b810194b-c717-4bf4-8558-07eab131bdc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116507d-hx69n" Oct 30 00:29:54.738580 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:54.738537 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4kdm\" (UniqueName: \"kubernetes.io/projected/b810194b-c717-4bf4-8558-07eab131bdc5-kube-api-access-h4kdm\") pod \"maas-api-revoke-sub-7116507d-hx69n\" (UID: \"b810194b-c717-4bf4-8558-07eab131bdc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116507d-hx69n" Oct 30 00:29:54.739195 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:54.739170 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b810194b-c717-4bf4-8558-07eab131bdc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7116507d-hx69n\" (UID: \"b810194b-c717-4bf4-8558-07eab131bdc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116507d-hx69n" Oct 30 00:29:54.740757 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:54.740735 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b810194b-c717-4bf4-8558-07eab131bdc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7116507d-hx69n\" (UID: \"b810194b-c717-4bf4-8558-07eab131bdc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116507d-hx69n" Oct 30 00:29:54.758727 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:54.758676 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4kdm\" (UniqueName: \"kubernetes.io/projected/b810194b-c717-4bf4-8558-07eab131bdc5-kube-api-access-h4kdm\") pod \"maas-api-revoke-sub-7116507d-hx69n\" (UID: \"b810194b-c717-4bf4-8558-07eab131bdc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116507d-hx69n" Oct 30 00:29:54.838529 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:54.838488 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116507d-hx69n" Oct 30 00:29:54.925072 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:54.925039 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91eb024d-d850-4682-9821-bcae4cb40b49" path="/var/lib/kubelet/pods/91eb024d-d850-4682-9821-bcae4cb40b49/volumes" Oct 30 00:29:55.054748 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:55.054711 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3536f0d4-tz9pw" event={"ID":"4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838","Type":"ContainerStarted","Data":"586b8ff2639d4721d508befbfa08b0229036a480297c281591653a54547104b0"} Oct 30 00:29:55.055200 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:55.054775 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3536f0d4-tz9pw" event={"ID":"4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838","Type":"ContainerStarted","Data":"e3737404aa5562ee2bef73b9517080010368fa23db49e14be858768a3fb05c04"} Oct 30 00:29:55.090576 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:55.090509 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3536f0d4-tz9pw" podStartSLOduration=2.090492038 podStartE2EDuration="2.090492038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:29:55.088899463 +0000 UTC m=+1510.294400645" watchObservedRunningTime="2026-10-30 00:29:55.090492038 +0000 UTC m=+1510.295993221" Oct 30 00:29:55.143794 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:55.143766 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7116507d-hx69n"] Oct 30 00:29:55.198953 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:29:55.198901 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb810194b_c717_4bf4_8558_07eab131bdc5.slice/crio-d715def268c3c328b20f62d1918478aaeb0352e2bc496c333076f110453850b7 WatchSource:0}: Error finding container d715def268c3c328b20f62d1918478aaeb0352e2bc496c333076f110453850b7: Status 404 returned error can't find the container with id d715def268c3c328b20f62d1918478aaeb0352e2bc496c333076f110453850b7 Oct 30 00:29:56.077054 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:56.077022 5001 generic.go:356] "Generic (PLEG): container finished" podID="4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838" containerID="586b8ff2639d4721d508befbfa08b0229036a480297c281591653a54547104b0" exitCode=0 Oct 30 00:29:56.077436 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:56.077109 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3536f0d4-tz9pw" event={"ID":"4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838","Type":"ContainerDied","Data":"586b8ff2639d4721d508befbfa08b0229036a480297c281591653a54547104b0"} Oct 30 00:29:56.078859 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:56.078829 5001 generic.go:356] "Generic (PLEG): container finished" podID="b810194b-c717-4bf4-8558-07eab131bdc5" containerID="840ccaea6e91f6ee8a6e7a21dc1e73379fa16c47e6a6392e15e41759be12012f" exitCode=0 Oct 30 00:29:56.078991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:56.078923 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116507d-hx69n" event={"ID":"b810194b-c717-4bf4-8558-07eab131bdc5","Type":"ContainerDied","Data":"840ccaea6e91f6ee8a6e7a21dc1e73379fa16c47e6a6392e15e41759be12012f"} Oct 30 00:29:56.078991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:56.078969 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116507d-hx69n" event={"ID":"b810194b-c717-4bf4-8558-07eab131bdc5","Type":"ContainerStarted","Data":"d715def268c3c328b20f62d1918478aaeb0352e2bc496c333076f110453850b7"} Oct 30 00:29:57.392813 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:57.392780 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3536f0d4-tz9pw" Oct 30 00:29:57.419181 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:57.419150 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116507d-hx69n" Oct 30 00:29:57.483744 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:57.483698 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838-kube-api-access-858bh\" (UniqueName: \"kubernetes.io/projected/4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838-kube-api-access-858bh\") pod \"4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838\" (UID: \"4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838\") " Oct 30 00:29:57.483986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:57.483794 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838-maas-api-openshift-ca\") pod \"4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838\" (UID: \"4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838\") " Oct 30 00:29:57.483986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:57.483828 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838-maas-api-xks-ca\") pod \"4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838\" (UID: \"4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838\") " Oct 30 00:29:57.483986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:57.483869 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b810194b-c717-4bf4-8558-07eab131bdc5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b810194b-c717-4bf4-8558-07eab131bdc5-maas-api-xks-ca\") pod \"b810194b-c717-4bf4-8558-07eab131bdc5\" (UID: \"b810194b-c717-4bf4-8558-07eab131bdc5\") " Oct 30 00:29:57.484200 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:57.484125 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b810194b-c717-4bf4-8558-07eab131bdc5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b810194b-c717-4bf4-8558-07eab131bdc5-maas-api-openshift-ca\") pod \"b810194b-c717-4bf4-8558-07eab131bdc5\" (UID: \"b810194b-c717-4bf4-8558-07eab131bdc5\") " Oct 30 00:29:57.484200 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:57.484167 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838-maas-api-openshift-ca" pod "4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838" (UID: "4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:29:57.484379 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:57.484186 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b810194b-c717-4bf4-8558-07eab131bdc5-kube-api-access-h4kdm\" (UniqueName: \"kubernetes.io/projected/b810194b-c717-4bf4-8558-07eab131bdc5-kube-api-access-h4kdm\") pod \"b810194b-c717-4bf4-8558-07eab131bdc5\" (UID: \"b810194b-c717-4bf4-8558-07eab131bdc5\") " Oct 30 00:29:57.484803 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:57.484774 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b810194b-c717-4bf4-8558-07eab131bdc5-maas-api-openshift-ca" pod "b810194b-c717-4bf4-8558-07eab131bdc5" (UID: "b810194b-c717-4bf4-8558-07eab131bdc5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:29:57.484984 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:57.484956 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b810194b-c717-4bf4-8558-07eab131bdc5-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:57.485079 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:57.484983 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:57.486252 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:57.486227 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838-kube-api-access-858bh" pod "4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838" (UID: "4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838"). InnerVolumeSpecName "kube-api-access-858bh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:29:57.486371 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:57.486354 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b810194b-c717-4bf4-8558-07eab131bdc5-kube-api-access-h4kdm" pod "b810194b-c717-4bf4-8558-07eab131bdc5" (UID: "b810194b-c717-4bf4-8558-07eab131bdc5"). InnerVolumeSpecName "kube-api-access-h4kdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:29:57.486474 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:57.486456 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b810194b-c717-4bf4-8558-07eab131bdc5-maas-api-xks-ca" pod "b810194b-c717-4bf4-8558-07eab131bdc5" (UID: "b810194b-c717-4bf4-8558-07eab131bdc5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:29:57.486516 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:57.486469 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838-maas-api-xks-ca" pod "4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838" (UID: "4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:29:57.528327 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:57.528299 5001 scope.go:122] "RemoveContainer" containerID="ecbcb624b7cb9386becb98fdaa8765a7651b4ebd4ed2758d290f9093295342d0" Oct 30 00:29:57.543420 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:57.541413 5001 scope.go:122] "RemoveContainer" containerID="8c979b1c5d79f2438c6cb7eebc6e1fd200d566c98770fbba03330199a07f18f0" Oct 30 00:29:57.552812 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:57.552788 5001 scope.go:122] "RemoveContainer" containerID="b55086178e9df08afa9b5f3f8f9bcc7064cfe69780845e5e5ba5b6d644613f4e" Oct 30 00:29:57.563158 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:57.563137 5001 scope.go:122] "RemoveContainer" containerID="c7495ab4a281a4d6591fbd11a8881bde1fb823476af06068a82a492871c8a673" Oct 30 00:29:57.586017 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:57.585994 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-858bh\" (UniqueName: \"kubernetes.io/projected/4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838-kube-api-access-858bh\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:57.586108 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:57.586022 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:57.586108 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:57.586037 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b810194b-c717-4bf4-8558-07eab131bdc5-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:57.586108 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:57.586049 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4kdm\" (UniqueName: \"kubernetes.io/projected/b810194b-c717-4bf4-8558-07eab131bdc5-kube-api-access-h4kdm\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:29:58.097763 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:58.097736 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3536f0d4-tz9pw" Oct 30 00:29:58.097974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:58.097811 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3536f0d4-tz9pw" event={"ID":"4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838","Type":"ContainerDied","Data":"e3737404aa5562ee2bef73b9517080010368fa23db49e14be858768a3fb05c04"} Oct 30 00:29:58.097974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:58.097860 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3737404aa5562ee2bef73b9517080010368fa23db49e14be858768a3fb05c04" Oct 30 00:29:58.099501 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:58.099481 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116507d-hx69n" Oct 30 00:29:58.099628 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:58.099513 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116507d-hx69n" event={"ID":"b810194b-c717-4bf4-8558-07eab131bdc5","Type":"ContainerDied","Data":"d715def268c3c328b20f62d1918478aaeb0352e2bc496c333076f110453850b7"} Oct 30 00:29:58.099628 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:58.099557 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d715def268c3c328b20f62d1918478aaeb0352e2bc496c333076f110453850b7" Oct 30 00:29:59.174173 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:59.174146 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f649b0-57947cf4fd-qdghs_4925fd35-f8b1-4706-990d-a291bb1ed760/payload-processing/0.log" Oct 30 00:29:59.374296 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:29:59.374268 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f649b0-57947cf4fd-qdghs_4925fd35-f8b1-4706-990d-a291bb1ed760/payload-processing/0.log" Oct 30 00:30:00.040935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.028228 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb3ad-6f6bb"] Oct 30 00:30:00.054935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.049986 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838" containerName="revoke-keys" Oct 30 00:30:00.054935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.050024 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838" containerName="revoke-keys" Oct 30 00:30:00.054935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.050098 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b810194b-c717-4bf4-8558-07eab131bdc5" containerName="revoke-keys" Oct 30 00:30:00.054935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.050107 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b810194b-c717-4bf4-8558-07eab131bdc5" containerName="revoke-keys" Oct 30 00:30:00.054935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.050412 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b810194b-c717-4bf4-8558-07eab131bdc5" containerName="revoke-keys" Oct 30 00:30:00.054935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.050443 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838" containerName="revoke-keys" Oct 30 00:30:00.058934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.057266 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb3ad-6f6bb" Oct 30 00:30:00.079427 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.070286 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:30:00.079427 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.070466 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:30:00.086177 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.081111 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:30:00.086177 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.081202 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:30:00.216396 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.216127 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb3ad-6f6bb"] Oct 30 00:30:00.225861 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.225772 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7"] Oct 30 00:30:00.268251 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.244070 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d35cbc6-e2c5-43a5-9749-3785f2dfdde1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fb3ad-6f6bb\" (UID: \"3d35cbc6-e2c5-43a5-9749-3785f2dfdde1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb3ad-6f6bb" Oct 30 00:30:00.430112 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.430021 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d35cbc6-e2c5-43a5-9749-3785f2dfdde1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fb3ad-6f6bb\" (UID: \"3d35cbc6-e2c5-43a5-9749-3785f2dfdde1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb3ad-6f6bb" Oct 30 00:30:00.430465 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.430440 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrrmr\" (UniqueName: \"kubernetes.io/projected/3d35cbc6-e2c5-43a5-9749-3785f2dfdde1-kube-api-access-zrrmr\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fb3ad-6f6bb\" (UID: \"3d35cbc6-e2c5-43a5-9749-3785f2dfdde1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb3ad-6f6bb" Oct 30 00:30:00.459680 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.459646 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6"] Oct 30 00:30:00.462740 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.462715 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7" Oct 30 00:30:00.469188 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.469163 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf"] Oct 30 00:30:00.474773 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.474751 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888670-279bh"] Oct 30 00:30:00.498937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.484663 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888670-279bh" Oct 30 00:30:00.525936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.499581 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6" Oct 30 00:30:00.541987 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.528742 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf" Oct 30 00:30:00.542378 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.536893 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d35cbc6-e2c5-43a5-9749-3785f2dfdde1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fb3ad-6f6bb\" (UID: \"3d35cbc6-e2c5-43a5-9749-3785f2dfdde1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb3ad-6f6bb" Oct 30 00:30:00.542827 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.542807 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrrmr\" (UniqueName: \"kubernetes.io/projected/3d35cbc6-e2c5-43a5-9749-3785f2dfdde1-kube-api-access-zrrmr\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fb3ad-6f6bb\" (UID: \"3d35cbc6-e2c5-43a5-9749-3785f2dfdde1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb3ad-6f6bb" Oct 30 00:30:00.545114 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.544137 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d35cbc6-e2c5-43a5-9749-3785f2dfdde1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fb3ad-6f6bb\" (UID: \"3d35cbc6-e2c5-43a5-9749-3785f2dfdde1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb3ad-6f6bb" Oct 30 00:30:00.545603 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.545070 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d35cbc6-e2c5-43a5-9749-3785f2dfdde1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fb3ad-6f6bb\" (UID: \"3d35cbc6-e2c5-43a5-9749-3785f2dfdde1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb3ad-6f6bb" Oct 30 00:30:00.545840 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.539854 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d35cbc6-e2c5-43a5-9749-3785f2dfdde1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fb3ad-6f6bb\" (UID: \"3d35cbc6-e2c5-43a5-9749-3785f2dfdde1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb3ad-6f6bb" Oct 30 00:30:00.547827 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.547806 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt"] Oct 30 00:30:00.569888 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.569401 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr"] Oct 30 00:30:00.569888 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.569642 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt" Oct 30 00:30:00.577931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.577855 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrrmr\" (UniqueName: \"kubernetes.io/projected/3d35cbc6-e2c5-43a5-9749-3785f2dfdde1-kube-api-access-zrrmr\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fb3ad-6f6bb\" (UID: \"3d35cbc6-e2c5-43a5-9749-3785f2dfdde1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb3ad-6f6bb" Oct 30 00:30:00.591550 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.591523 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr" Oct 30 00:30:00.591730 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.591567 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2"] Oct 30 00:30:00.597862 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.597833 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7"] Oct 30 00:30:00.598040 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.597895 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6"] Oct 30 00:30:00.598040 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.597955 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2" Oct 30 00:30:00.598040 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.597972 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf"] Oct 30 00:30:00.598040 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.597993 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9"] Oct 30 00:30:00.604321 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.604295 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr"] Oct 30 00:30:00.604471 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.604412 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9" Oct 30 00:30:00.616237 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.616200 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2"] Oct 30 00:30:00.631240 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.624608 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888670-279bh"] Oct 30 00:30:00.641524 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.639187 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt"] Oct 30 00:30:00.662463 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.662083 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55r8g\" (UniqueName: \"kubernetes.io/projected/2c07d333-2dda-47eb-81e0-4b99d2fffc85-kube-api-access-55r8g\") pod \"maas-api-key-cleanup-29888670-279bh\" (UID: \"2c07d333-2dda-47eb-81e0-4b99d2fffc85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888670-279bh" Oct 30 00:30:00.666314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.662732 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpzfd\" (UniqueName: \"kubernetes.io/projected/2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d-kube-api-access-mpzfd\") pod \"maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7\" (UID: \"2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7" Oct 30 00:30:00.666314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.662827 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr529\" (UniqueName: \"kubernetes.io/projected/42f34b56-b044-4973-a43d-6e8e3b8050a9-kube-api-access-qr529\") pod \"maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf\" (UID: \"42f34b56-b044-4973-a43d-6e8e3b8050a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf" Oct 30 00:30:00.666314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.662857 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d09f85-579f-4726-95a9-b6680576db12-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr\" (UID: \"d0d09f85-579f-4726-95a9-b6680576db12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr" Oct 30 00:30:00.666314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.662893 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de5e2310-d7f8-45df-8613-6977a1ef3866-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6\" (UID: \"de5e2310-d7f8-45df-8613-6977a1ef3866\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6" Oct 30 00:30:00.666314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.662999 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7\" (UID: \"2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7" Oct 30 00:30:00.666314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.663035 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fba3789-18ae-409a-bdf4-8e47e68a46ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9\" (UID: \"4fba3789-18ae-409a-bdf4-8e47e68a46ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9" Oct 30 00:30:00.666314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.663078 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42f34b56-b044-4973-a43d-6e8e3b8050a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf\" (UID: \"42f34b56-b044-4973-a43d-6e8e3b8050a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf" Oct 30 00:30:00.666314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.663159 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cd88\" (UniqueName: \"kubernetes.io/projected/d0d09f85-579f-4726-95a9-b6680576db12-kube-api-access-7cd88\") pod \"maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr\" (UID: \"d0d09f85-579f-4726-95a9-b6680576db12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr" Oct 30 00:30:00.666314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.663193 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2npd\" (UniqueName: \"kubernetes.io/projected/3e472400-6cd4-4ef4-a999-5717b9570164-kube-api-access-m2npd\") pod \"maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2\" (UID: \"3e472400-6cd4-4ef4-a999-5717b9570164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2" Oct 30 00:30:00.666314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.663224 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6aa747-bb02-4dda-8570-745914ae491f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt\" (UID: \"dd6aa747-bb02-4dda-8570-745914ae491f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt" Oct 30 00:30:00.666314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.663258 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nlvt\" (UniqueName: \"kubernetes.io/projected/4fba3789-18ae-409a-bdf4-8e47e68a46ba-kube-api-access-6nlvt\") pod \"maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9\" (UID: \"4fba3789-18ae-409a-bdf4-8e47e68a46ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9" Oct 30 00:30:00.666314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.663346 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lgnz\" (UniqueName: \"kubernetes.io/projected/de5e2310-d7f8-45df-8613-6977a1ef3866-kube-api-access-8lgnz\") pod \"maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6\" (UID: \"de5e2310-d7f8-45df-8613-6977a1ef3866\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6" Oct 30 00:30:00.666314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.663422 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c07d333-2dda-47eb-81e0-4b99d2fffc85-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888670-279bh\" (UID: \"2c07d333-2dda-47eb-81e0-4b99d2fffc85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888670-279bh" Oct 30 00:30:00.666314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.663497 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e472400-6cd4-4ef4-a999-5717b9570164-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2\" (UID: \"3e472400-6cd4-4ef4-a999-5717b9570164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2" Oct 30 00:30:00.666314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.663548 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbnw5\" (UniqueName: \"kubernetes.io/projected/dd6aa747-bb02-4dda-8570-745914ae491f-kube-api-access-qbnw5\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt\" (UID: \"dd6aa747-bb02-4dda-8570-745914ae491f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt" Oct 30 00:30:00.706935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.700200 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9"] Oct 30 00:30:00.754396 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.754330 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43dc72-mw75l"] Oct 30 00:30:00.759664 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.759641 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43dc72-mw75l"] Oct 30 00:30:00.764501 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.764471 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbnw5\" (UniqueName: \"kubernetes.io/projected/dd6aa747-bb02-4dda-8570-745914ae491f-kube-api-access-qbnw5\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt\" (UID: \"dd6aa747-bb02-4dda-8570-745914ae491f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt" Oct 30 00:30:00.764628 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.764546 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55r8g\" (UniqueName: \"kubernetes.io/projected/2c07d333-2dda-47eb-81e0-4b99d2fffc85-kube-api-access-55r8g\") pod \"maas-api-key-cleanup-29888670-279bh\" (UID: \"2c07d333-2dda-47eb-81e0-4b99d2fffc85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888670-279bh" Oct 30 00:30:00.764628 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.764589 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpzfd\" (UniqueName: \"kubernetes.io/projected/2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d-kube-api-access-mpzfd\") pod \"maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7\" (UID: \"2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7" Oct 30 00:30:00.764628 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.764624 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr529\" (UniqueName: \"kubernetes.io/projected/42f34b56-b044-4973-a43d-6e8e3b8050a9-kube-api-access-qr529\") pod \"maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf\" (UID: \"42f34b56-b044-4973-a43d-6e8e3b8050a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf" Oct 30 00:30:00.764799 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.764649 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d09f85-579f-4726-95a9-b6680576db12-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr\" (UID: \"d0d09f85-579f-4726-95a9-b6680576db12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr" Oct 30 00:30:00.764799 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.764678 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de5e2310-d7f8-45df-8613-6977a1ef3866-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6\" (UID: \"de5e2310-d7f8-45df-8613-6977a1ef3866\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6" Oct 30 00:30:00.764799 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.764756 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7\" (UID: \"2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7" Oct 30 00:30:00.764799 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.764788 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fba3789-18ae-409a-bdf4-8e47e68a46ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9\" (UID: \"4fba3789-18ae-409a-bdf4-8e47e68a46ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9" Oct 30 00:30:00.765246 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.764831 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42f34b56-b044-4973-a43d-6e8e3b8050a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf\" (UID: \"42f34b56-b044-4973-a43d-6e8e3b8050a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf" Oct 30 00:30:00.765246 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.765088 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cd88\" (UniqueName: \"kubernetes.io/projected/d0d09f85-579f-4726-95a9-b6680576db12-kube-api-access-7cd88\") pod \"maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr\" (UID: \"d0d09f85-579f-4726-95a9-b6680576db12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr" Oct 30 00:30:00.765246 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.765135 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2npd\" (UniqueName: \"kubernetes.io/projected/3e472400-6cd4-4ef4-a999-5717b9570164-kube-api-access-m2npd\") pod \"maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2\" (UID: \"3e472400-6cd4-4ef4-a999-5717b9570164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2" Oct 30 00:30:00.765246 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.765166 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6aa747-bb02-4dda-8570-745914ae491f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt\" (UID: \"dd6aa747-bb02-4dda-8570-745914ae491f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt" Oct 30 00:30:00.765246 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.765199 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nlvt\" (UniqueName: \"kubernetes.io/projected/4fba3789-18ae-409a-bdf4-8e47e68a46ba-kube-api-access-6nlvt\") pod \"maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9\" (UID: \"4fba3789-18ae-409a-bdf4-8e47e68a46ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9" Oct 30 00:30:00.765534 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.765288 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lgnz\" (UniqueName: \"kubernetes.io/projected/de5e2310-d7f8-45df-8613-6977a1ef3866-kube-api-access-8lgnz\") pod \"maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6\" (UID: \"de5e2310-d7f8-45df-8613-6977a1ef3866\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6" Oct 30 00:30:00.765534 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.765362 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c07d333-2dda-47eb-81e0-4b99d2fffc85-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888670-279bh\" (UID: \"2c07d333-2dda-47eb-81e0-4b99d2fffc85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888670-279bh" Oct 30 00:30:00.765534 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.765435 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e472400-6cd4-4ef4-a999-5717b9570164-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2\" (UID: \"3e472400-6cd4-4ef4-a999-5717b9570164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2" Oct 30 00:30:00.765716 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.765608 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7\" (UID: \"2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7" Oct 30 00:30:00.765716 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.765678 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d09f85-579f-4726-95a9-b6680576db12-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr\" (UID: \"d0d09f85-579f-4726-95a9-b6680576db12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr" Oct 30 00:30:00.765716 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.765608 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de5e2310-d7f8-45df-8613-6977a1ef3866-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6\" (UID: \"de5e2310-d7f8-45df-8613-6977a1ef3866\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6" Oct 30 00:30:00.765883 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.765818 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fba3789-18ae-409a-bdf4-8e47e68a46ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9\" (UID: \"4fba3789-18ae-409a-bdf4-8e47e68a46ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9" Oct 30 00:30:00.766326 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.766299 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42f34b56-b044-4973-a43d-6e8e3b8050a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf\" (UID: \"42f34b56-b044-4973-a43d-6e8e3b8050a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf" Oct 30 00:30:00.766431 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.766385 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e472400-6cd4-4ef4-a999-5717b9570164-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2\" (UID: \"3e472400-6cd4-4ef4-a999-5717b9570164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2" Oct 30 00:30:00.766519 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.766495 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6aa747-bb02-4dda-8570-745914ae491f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt\" (UID: \"dd6aa747-bb02-4dda-8570-745914ae491f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt" Oct 30 00:30:00.766582 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.766562 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c07d333-2dda-47eb-81e0-4b99d2fffc85-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888670-279bh\" (UID: \"2c07d333-2dda-47eb-81e0-4b99d2fffc85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888670-279bh" Oct 30 00:30:00.795971 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.795945 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb3ad-6f6bb" Oct 30 00:30:00.819146 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.819120 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2npd\" (UniqueName: \"kubernetes.io/projected/3e472400-6cd4-4ef4-a999-5717b9570164-kube-api-access-m2npd\") pod \"maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2\" (UID: \"3e472400-6cd4-4ef4-a999-5717b9570164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2" Oct 30 00:30:00.819343 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.819310 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nlvt\" (UniqueName: \"kubernetes.io/projected/4fba3789-18ae-409a-bdf4-8e47e68a46ba-kube-api-access-6nlvt\") pod \"maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9\" (UID: \"4fba3789-18ae-409a-bdf4-8e47e68a46ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9" Oct 30 00:30:00.821951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.820009 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbnw5\" (UniqueName: \"kubernetes.io/projected/dd6aa747-bb02-4dda-8570-745914ae491f-kube-api-access-qbnw5\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt\" (UID: \"dd6aa747-bb02-4dda-8570-745914ae491f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt" Oct 30 00:30:00.821951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.820448 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpzfd\" (UniqueName: \"kubernetes.io/projected/2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d-kube-api-access-mpzfd\") pod \"maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7\" (UID: \"2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7" Oct 30 00:30:00.822532 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.822501 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lgnz\" (UniqueName: \"kubernetes.io/projected/de5e2310-d7f8-45df-8613-6977a1ef3866-kube-api-access-8lgnz\") pod \"maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6\" (UID: \"de5e2310-d7f8-45df-8613-6977a1ef3866\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6" Oct 30 00:30:00.822718 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.822700 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr529\" (UniqueName: \"kubernetes.io/projected/42f34b56-b044-4973-a43d-6e8e3b8050a9-kube-api-access-qr529\") pod \"maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf\" (UID: \"42f34b56-b044-4973-a43d-6e8e3b8050a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf" Oct 30 00:30:00.823582 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.823563 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cd88\" (UniqueName: \"kubernetes.io/projected/d0d09f85-579f-4726-95a9-b6680576db12-kube-api-access-7cd88\") pod \"maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr\" (UID: \"d0d09f85-579f-4726-95a9-b6680576db12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr" Oct 30 00:30:00.825409 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.825382 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55r8g\" (UniqueName: \"kubernetes.io/projected/2c07d333-2dda-47eb-81e0-4b99d2fffc85-kube-api-access-55r8g\") pod \"maas-api-key-cleanup-29888670-279bh\" (UID: \"2c07d333-2dda-47eb-81e0-4b99d2fffc85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888670-279bh" Oct 30 00:30:00.921346 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.921311 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7ee16fd-d8bc-4aee-b956-62168a6e0fdb" path="/var/lib/kubelet/pods/e7ee16fd-d8bc-4aee-b956-62168a6e0fdb/volumes" Oct 30 00:30:00.943254 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.943186 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7" Oct 30 00:30:00.990751 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:00.990717 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6" Oct 30 00:30:01.003145 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:01.002852 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888670-279bh" Oct 30 00:30:01.057415 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:01.057360 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf" Oct 30 00:30:01.062968 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:01.062921 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt" Oct 30 00:30:01.074081 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:01.074046 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr" Oct 30 00:30:01.082987 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:01.082661 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2" Oct 30 00:30:01.088958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:01.088639 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9" Oct 30 00:30:01.117387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:01.116865 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb3ad-6f6bb"] Oct 30 00:30:01.130673 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:30:01.130549 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d35cbc6_e2c5_43a5_9749_3785f2dfdde1.slice/crio-e4dcaa73048517048457adcf236ce13014574f002869b80e0165d0602083822c WatchSource:0}: Error finding container e4dcaa73048517048457adcf236ce13014574f002869b80e0165d0602083822c: Status 404 returned error can't find the container with id e4dcaa73048517048457adcf236ce13014574f002869b80e0165d0602083822c Oct 30 00:30:01.261351 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:01.260609 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7"] Oct 30 00:30:01.342151 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:01.342056 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb3ad-6f6bb" event={"ID":"3d35cbc6-e2c5-43a5-9749-3785f2dfdde1","Type":"ContainerStarted","Data":"e4dcaa73048517048457adcf236ce13014574f002869b80e0165d0602083822c"} Oct 30 00:30:01.345173 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:01.345093 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7" event={"ID":"2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d","Type":"ContainerStarted","Data":"525c74a0861c2f3c7d00ca7a0fe7d955ed876f0131e2e28092fe59c4ad77d278"} Oct 30 00:30:01.384055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:01.384024 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6"] Oct 30 00:30:01.395281 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:01.395253 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888670-279bh"] Oct 30 00:30:01.539110 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:01.539040 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2"] Oct 30 00:30:01.546166 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:01.546141 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr"] Oct 30 00:30:01.546448 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:30:01.546417 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d09f85_579f_4726_95a9_b6680576db12.slice/crio-e863616a70f983711b8f7ff8b8f59b913b3ddda324819c7ee8a69b95a533953c WatchSource:0}: Error finding container e863616a70f983711b8f7ff8b8f59b913b3ddda324819c7ee8a69b95a533953c: Status 404 returned error can't find the container with id e863616a70f983711b8f7ff8b8f59b913b3ddda324819c7ee8a69b95a533953c Oct 30 00:30:01.551249 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:30:01.551225 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd6aa747_bb02_4dda_8570_745914ae491f.slice/crio-4f62b134bd66c25ad546d398c36e9af240cf39abe6c693c7a0f50d229ae357b7 WatchSource:0}: Error finding container 4f62b134bd66c25ad546d398c36e9af240cf39abe6c693c7a0f50d229ae357b7: Status 404 returned error can't find the container with id 4f62b134bd66c25ad546d398c36e9af240cf39abe6c693c7a0f50d229ae357b7 Oct 30 00:30:01.552133 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:01.552110 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt"] Oct 30 00:30:01.564875 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:01.564849 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9"] Oct 30 00:30:01.566871 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:30:01.566843 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fba3789_18ae_409a_bdf4_8e47e68a46ba.slice/crio-f9df4acbae401f49fb94239724c0bfe2dce61ae17c720c296391f93394b9c3e2 WatchSource:0}: Error finding container f9df4acbae401f49fb94239724c0bfe2dce61ae17c720c296391f93394b9c3e2: Status 404 returned error can't find the container with id f9df4acbae401f49fb94239724c0bfe2dce61ae17c720c296391f93394b9c3e2 Oct 30 00:30:01.646556 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:01.646523 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf"] Oct 30 00:30:01.712487 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:30:01.712451 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f34b56_b044_4973_a43d_6e8e3b8050a9.slice/crio-41a0f8c18652878a038698e52b3599c2bd91fa760f62ded545f8f2be6c0ef3af WatchSource:0}: Error finding container 41a0f8c18652878a038698e52b3599c2bd91fa760f62ded545f8f2be6c0ef3af: Status 404 returned error can't find the container with id 41a0f8c18652878a038698e52b3599c2bd91fa760f62ded545f8f2be6c0ef3af Oct 30 00:30:02.355688 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:02.355663 5001 generic.go:356] "Generic (PLEG): container finished" podID="3d35cbc6-e2c5-43a5-9749-3785f2dfdde1" containerID="f95650d02898872208e7c752adf838a48a21b56c427f7641368615c92d74ca13" exitCode=0 Oct 30 00:30:02.356036 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:02.355738 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb3ad-6f6bb" event={"ID":"3d35cbc6-e2c5-43a5-9749-3785f2dfdde1","Type":"ContainerDied","Data":"f95650d02898872208e7c752adf838a48a21b56c427f7641368615c92d74ca13"} Oct 30 00:30:02.357185 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:02.357161 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr" event={"ID":"d0d09f85-579f-4726-95a9-b6680576db12","Type":"ContainerStarted","Data":"e863616a70f983711b8f7ff8b8f59b913b3ddda324819c7ee8a69b95a533953c"} Oct 30 00:30:02.358672 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:02.358642 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf" event={"ID":"42f34b56-b044-4973-a43d-6e8e3b8050a9","Type":"ContainerStarted","Data":"8e4f7cce75a36f2a44d1e73c4d2e122eb647f7e763a12f10fb7ec1816b876a55"} Oct 30 00:30:02.358772 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:02.358687 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf" event={"ID":"42f34b56-b044-4973-a43d-6e8e3b8050a9","Type":"ContainerStarted","Data":"41a0f8c18652878a038698e52b3599c2bd91fa760f62ded545f8f2be6c0ef3af"} Oct 30 00:30:02.360248 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:02.360224 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888670-279bh" event={"ID":"2c07d333-2dda-47eb-81e0-4b99d2fffc85","Type":"ContainerStarted","Data":"a4e1c6afcd4a3ad5dea1f770efc6b8c25c84bbd7f7ac6c69f79af6146315a38b"} Oct 30 00:30:02.360365 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:02.360268 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888670-279bh" event={"ID":"2c07d333-2dda-47eb-81e0-4b99d2fffc85","Type":"ContainerStarted","Data":"be165850175dd8724cba0d18539a31bacf4933db1641f6a7efc9dd84725e4a0c"} Oct 30 00:30:02.361921 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:02.361882 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2" event={"ID":"3e472400-6cd4-4ef4-a999-5717b9570164","Type":"ContainerStarted","Data":"85e8aee40f8f295ade83b3b38efc7f506c16125d0a7cb32272208db2a3280422"} Oct 30 00:30:02.362030 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:02.361975 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2" event={"ID":"3e472400-6cd4-4ef4-a999-5717b9570164","Type":"ContainerStarted","Data":"8aaeb43dc6279634f5d777cb45ae483cf59d802eba1fa78f15b9f8b65508096e"} Oct 30 00:30:02.363465 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:02.363445 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt" event={"ID":"dd6aa747-bb02-4dda-8570-745914ae491f","Type":"ContainerStarted","Data":"4b26e614d67c35f0120eadebcfb4abea531b2a964409e9d31d41386110c84e59"} Oct 30 00:30:02.363560 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:02.363485 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt" event={"ID":"dd6aa747-bb02-4dda-8570-745914ae491f","Type":"ContainerStarted","Data":"4f62b134bd66c25ad546d398c36e9af240cf39abe6c693c7a0f50d229ae357b7"} Oct 30 00:30:02.365018 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:02.364996 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6" event={"ID":"de5e2310-d7f8-45df-8613-6977a1ef3866","Type":"ContainerStarted","Data":"49aed6fa076b5503ddcf409ca5b03f6b795670287e25f57ceba8016272c3d2ec"} Oct 30 00:30:02.365132 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:02.365030 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6" event={"ID":"de5e2310-d7f8-45df-8613-6977a1ef3866","Type":"ContainerStarted","Data":"10379fa518783248a70e509f22bce9ff57aae861eefcd500b8aefe390d42344c"} Oct 30 00:30:02.366524 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:02.366505 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7" event={"ID":"2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d","Type":"ContainerStarted","Data":"8700bafb7b23777de0ee42fd3804b1e6c048684cdbd6fe2a884cadcc5e44af44"} Oct 30 00:30:02.368166 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:02.368115 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9" event={"ID":"4fba3789-18ae-409a-bdf4-8e47e68a46ba","Type":"ContainerStarted","Data":"1b89e4d88536e0260a2bfc04195455eda8e0007b49947ed7cf76bc7dee3cd624"} Oct 30 00:30:02.368166 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:02.368150 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9" event={"ID":"4fba3789-18ae-409a-bdf4-8e47e68a46ba","Type":"ContainerStarted","Data":"f9df4acbae401f49fb94239724c0bfe2dce61ae17c720c296391f93394b9c3e2"} Oct 30 00:30:02.430751 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:02.430669 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7" podStartSLOduration=2.430651288 podStartE2EDuration="2.430651288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:30:02.423093081 +0000 UTC m=+1517.628594262" watchObservedRunningTime="2026-10-30 00:30:02.430651288 +0000 UTC m=+1517.636152470" Oct 30 00:30:02.460358 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:02.460308 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2" podStartSLOduration=2.460294742 podStartE2EDuration="2.460294742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:30:02.45620036 +0000 UTC m=+1517.661701540" watchObservedRunningTime="2026-10-30 00:30:02.460294742 +0000 UTC m=+1517.665795922" Oct 30 00:30:02.490081 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:02.490042 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888670-279bh" podStartSLOduration=2.490030612 podStartE2EDuration="2.490030612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:30:02.48955154 +0000 UTC m=+1517.695052720" watchObservedRunningTime="2026-10-30 00:30:02.490030612 +0000 UTC m=+1517.695531792" Oct 30 00:30:02.529798 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:02.529747 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9" podStartSLOduration=2.529733957 podStartE2EDuration="2.529733957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:30:02.524949337 +0000 UTC m=+1517.730450521" watchObservedRunningTime="2026-10-30 00:30:02.529733957 +0000 UTC m=+1517.735235137" Oct 30 00:30:02.559813 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:02.559769 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt" podStartSLOduration=2.5597548039999998 podStartE2EDuration="2.559754804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:30:02.558169132 +0000 UTC m=+1517.763670334" watchObservedRunningTime="2026-10-30 00:30:02.559754804 +0000 UTC m=+1517.765255987" Oct 30 00:30:02.591813 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:02.591775 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf" podStartSLOduration=2.591757587 podStartE2EDuration="2.591757587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:30:02.591629283 +0000 UTC m=+1517.797130463" watchObservedRunningTime="2026-10-30 00:30:02.591757587 +0000 UTC m=+1517.797258770" Oct 30 00:30:02.627936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:02.627641 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6" podStartSLOduration=2.627622992 podStartE2EDuration="2.627622992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:30:02.625161294 +0000 UTC m=+1517.830662493" watchObservedRunningTime="2026-10-30 00:30:02.627622992 +0000 UTC m=+1517.833124173" Oct 30 00:30:03.380202 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.380167 5001 generic.go:356] "Generic (PLEG): container finished" podID="d0d09f85-579f-4726-95a9-b6680576db12" containerID="1c354f3c498fa796fbd1c05acb3d5ecf0ca2191d3251dbfd296a14e8daa61b34" exitCode=60 Oct 30 00:30:03.380677 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.380251 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr" event={"ID":"d0d09f85-579f-4726-95a9-b6680576db12","Type":"ContainerDied","Data":"1c354f3c498fa796fbd1c05acb3d5ecf0ca2191d3251dbfd296a14e8daa61b34"} Oct 30 00:30:03.380677 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.380558 5001 scope.go:122] "RemoveContainer" containerID="1c354f3c498fa796fbd1c05acb3d5ecf0ca2191d3251dbfd296a14e8daa61b34" Oct 30 00:30:03.382088 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.382065 5001 generic.go:356] "Generic (PLEG): container finished" podID="42f34b56-b044-4973-a43d-6e8e3b8050a9" containerID="8e4f7cce75a36f2a44d1e73c4d2e122eb647f7e763a12f10fb7ec1816b876a55" exitCode=60 Oct 30 00:30:03.382201 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.382133 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf" event={"ID":"42f34b56-b044-4973-a43d-6e8e3b8050a9","Type":"ContainerDied","Data":"8e4f7cce75a36f2a44d1e73c4d2e122eb647f7e763a12f10fb7ec1816b876a55"} Oct 30 00:30:03.382449 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.382421 5001 scope.go:122] "RemoveContainer" containerID="8e4f7cce75a36f2a44d1e73c4d2e122eb647f7e763a12f10fb7ec1816b876a55" Oct 30 00:30:03.384682 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.384322 5001 generic.go:356] "Generic (PLEG): container finished" podID="2c07d333-2dda-47eb-81e0-4b99d2fffc85" containerID="a4e1c6afcd4a3ad5dea1f770efc6b8c25c84bbd7f7ac6c69f79af6146315a38b" exitCode=60 Oct 30 00:30:03.384682 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.384629 5001 scope.go:122] "RemoveContainer" containerID="a4e1c6afcd4a3ad5dea1f770efc6b8c25c84bbd7f7ac6c69f79af6146315a38b" Oct 30 00:30:03.384805 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.384405 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888670-279bh" event={"ID":"2c07d333-2dda-47eb-81e0-4b99d2fffc85","Type":"ContainerDied","Data":"a4e1c6afcd4a3ad5dea1f770efc6b8c25c84bbd7f7ac6c69f79af6146315a38b"} Oct 30 00:30:03.386524 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.386502 5001 generic.go:356] "Generic (PLEG): container finished" podID="3e472400-6cd4-4ef4-a999-5717b9570164" containerID="85e8aee40f8f295ade83b3b38efc7f506c16125d0a7cb32272208db2a3280422" exitCode=60 Oct 30 00:30:03.386675 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.386586 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2" event={"ID":"3e472400-6cd4-4ef4-a999-5717b9570164","Type":"ContainerDied","Data":"85e8aee40f8f295ade83b3b38efc7f506c16125d0a7cb32272208db2a3280422"} Oct 30 00:30:03.386906 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.386888 5001 scope.go:122] "RemoveContainer" containerID="85e8aee40f8f295ade83b3b38efc7f506c16125d0a7cb32272208db2a3280422" Oct 30 00:30:03.389164 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.389139 5001 generic.go:356] "Generic (PLEG): container finished" podID="dd6aa747-bb02-4dda-8570-745914ae491f" containerID="4b26e614d67c35f0120eadebcfb4abea531b2a964409e9d31d41386110c84e59" exitCode=60 Oct 30 00:30:03.389283 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.389239 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt" event={"ID":"dd6aa747-bb02-4dda-8570-745914ae491f","Type":"ContainerDied","Data":"4b26e614d67c35f0120eadebcfb4abea531b2a964409e9d31d41386110c84e59"} Oct 30 00:30:03.389503 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.389483 5001 scope.go:122] "RemoveContainer" containerID="4b26e614d67c35f0120eadebcfb4abea531b2a964409e9d31d41386110c84e59" Oct 30 00:30:03.391015 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.390997 5001 generic.go:356] "Generic (PLEG): container finished" podID="de5e2310-d7f8-45df-8613-6977a1ef3866" containerID="49aed6fa076b5503ddcf409ca5b03f6b795670287e25f57ceba8016272c3d2ec" exitCode=60 Oct 30 00:30:03.391114 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.391089 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6" event={"ID":"de5e2310-d7f8-45df-8613-6977a1ef3866","Type":"ContainerDied","Data":"49aed6fa076b5503ddcf409ca5b03f6b795670287e25f57ceba8016272c3d2ec"} Oct 30 00:30:03.391252 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.391237 5001 scope.go:122] "RemoveContainer" containerID="49aed6fa076b5503ddcf409ca5b03f6b795670287e25f57ceba8016272c3d2ec" Oct 30 00:30:03.392755 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.392735 5001 generic.go:356] "Generic (PLEG): container finished" podID="2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d" containerID="8700bafb7b23777de0ee42fd3804b1e6c048684cdbd6fe2a884cadcc5e44af44" exitCode=60 Oct 30 00:30:03.392850 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.392837 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7" event={"ID":"2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d","Type":"ContainerDied","Data":"8700bafb7b23777de0ee42fd3804b1e6c048684cdbd6fe2a884cadcc5e44af44"} Oct 30 00:30:03.393092 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.393076 5001 scope.go:122] "RemoveContainer" containerID="8700bafb7b23777de0ee42fd3804b1e6c048684cdbd6fe2a884cadcc5e44af44" Oct 30 00:30:03.394872 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.394849 5001 generic.go:356] "Generic (PLEG): container finished" podID="4fba3789-18ae-409a-bdf4-8e47e68a46ba" containerID="1b89e4d88536e0260a2bfc04195455eda8e0007b49947ed7cf76bc7dee3cd624" exitCode=60 Oct 30 00:30:03.394993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.394959 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9" event={"ID":"4fba3789-18ae-409a-bdf4-8e47e68a46ba","Type":"ContainerDied","Data":"1b89e4d88536e0260a2bfc04195455eda8e0007b49947ed7cf76bc7dee3cd624"} Oct 30 00:30:03.395155 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.395143 5001 scope.go:122] "RemoveContainer" containerID="1b89e4d88536e0260a2bfc04195455eda8e0007b49947ed7cf76bc7dee3cd624" Oct 30 00:30:03.714496 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.714468 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb3ad-6f6bb" Oct 30 00:30:03.820052 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.820026 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d35cbc6-e2c5-43a5-9749-3785f2dfdde1-kube-api-access-zrrmr\" (UniqueName: \"kubernetes.io/projected/3d35cbc6-e2c5-43a5-9749-3785f2dfdde1-kube-api-access-zrrmr\") pod \"3d35cbc6-e2c5-43a5-9749-3785f2dfdde1\" (UID: \"3d35cbc6-e2c5-43a5-9749-3785f2dfdde1\") " Oct 30 00:30:03.820216 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.820184 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d35cbc6-e2c5-43a5-9749-3785f2dfdde1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d35cbc6-e2c5-43a5-9749-3785f2dfdde1-maas-api-openshift-ca\") pod \"3d35cbc6-e2c5-43a5-9749-3785f2dfdde1\" (UID: \"3d35cbc6-e2c5-43a5-9749-3785f2dfdde1\") " Oct 30 00:30:03.820377 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.820243 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d35cbc6-e2c5-43a5-9749-3785f2dfdde1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d35cbc6-e2c5-43a5-9749-3785f2dfdde1-maas-api-xks-ca\") pod \"3d35cbc6-e2c5-43a5-9749-3785f2dfdde1\" (UID: \"3d35cbc6-e2c5-43a5-9749-3785f2dfdde1\") " Oct 30 00:30:03.820960 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.820934 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d35cbc6-e2c5-43a5-9749-3785f2dfdde1-maas-api-openshift-ca" pod "3d35cbc6-e2c5-43a5-9749-3785f2dfdde1" (UID: "3d35cbc6-e2c5-43a5-9749-3785f2dfdde1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:30:03.916089 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.916053 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d35cbc6-e2c5-43a5-9749-3785f2dfdde1-kube-api-access-zrrmr" pod "3d35cbc6-e2c5-43a5-9749-3785f2dfdde1" (UID: "3d35cbc6-e2c5-43a5-9749-3785f2dfdde1"). InnerVolumeSpecName "kube-api-access-zrrmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:30:03.917694 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.917652 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d35cbc6-e2c5-43a5-9749-3785f2dfdde1-maas-api-xks-ca" pod "3d35cbc6-e2c5-43a5-9749-3785f2dfdde1" (UID: "3d35cbc6-e2c5-43a5-9749-3785f2dfdde1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:30:03.922517 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.922494 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrrmr\" (UniqueName: \"kubernetes.io/projected/3d35cbc6-e2c5-43a5-9749-3785f2dfdde1-kube-api-access-zrrmr\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:03.922639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.922522 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d35cbc6-e2c5-43a5-9749-3785f2dfdde1-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:03.922639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:03.922548 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d35cbc6-e2c5-43a5-9749-3785f2dfdde1-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:04.488050 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:04.486758 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr" event={"ID":"d0d09f85-579f-4726-95a9-b6680576db12","Type":"ContainerStarted","Data":"c5f86f12e7550adccf27510a6946ead0002ed45604d9f736aa3f10210f0a097a"} Oct 30 00:30:04.489712 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:04.489683 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf" event={"ID":"42f34b56-b044-4973-a43d-6e8e3b8050a9","Type":"ContainerStarted","Data":"d858bc2cfe1f066d105d56817423fea863ebf9d0e295f2210f4990cbc27655e3"} Oct 30 00:30:04.492325 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:04.492293 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888670-279bh" event={"ID":"2c07d333-2dda-47eb-81e0-4b99d2fffc85","Type":"ContainerStarted","Data":"6c82afc3852f08a68fa18b46d4428e1e8a4f5a7ef0aba8410c1c7dc42611f6e2"} Oct 30 00:30:04.495373 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:04.495348 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2" event={"ID":"3e472400-6cd4-4ef4-a999-5717b9570164","Type":"ContainerStarted","Data":"dc2e619d07e1539475f62f22a0f43e2b4ae72280bc5922d58425897e91e51513"} Oct 30 00:30:04.498799 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:04.498776 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt" event={"ID":"dd6aa747-bb02-4dda-8570-745914ae491f","Type":"ContainerStarted","Data":"7a272978608e817ddb69a288ddea927ebb1e5ac7c7585ac0abf1beeffb48f549"} Oct 30 00:30:04.510402 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:04.510372 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6" event={"ID":"de5e2310-d7f8-45df-8613-6977a1ef3866","Type":"ContainerStarted","Data":"7aebf6e7724a3d729b8301350aea8e80185b7ba7876fd88619b868f99177c361"} Oct 30 00:30:04.511056 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:04.511035 5001 scope.go:122] "RemoveContainer" containerID="7aebf6e7724a3d729b8301350aea8e80185b7ba7876fd88619b868f99177c361" Oct 30 00:30:04.511570 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:30:04.511544 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6_odh-ai-gateway-infra(de5e2310-d7f8-45df-8613-6977a1ef3866)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6" podUID="de5e2310-d7f8-45df-8613-6977a1ef3866" Oct 30 00:30:04.551728 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:04.551696 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7" event={"ID":"2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d","Type":"ContainerStarted","Data":"8a42e4779fabf528e4b89267215511026de2c6979d2c460d97f31c857243d07a"} Oct 30 00:30:04.551962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:04.551939 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9" event={"ID":"4fba3789-18ae-409a-bdf4-8e47e68a46ba","Type":"ContainerStarted","Data":"f7fca1a18cdb18a5098d1e22ad7fdf9c97c52c7ef82bddfd93e6361f14661707"} Oct 30 00:30:04.553863 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:04.553839 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb3ad-6f6bb" event={"ID":"3d35cbc6-e2c5-43a5-9749-3785f2dfdde1","Type":"ContainerDied","Data":"e4dcaa73048517048457adcf236ce13014574f002869b80e0165d0602083822c"} Oct 30 00:30:04.554048 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:04.554029 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4dcaa73048517048457adcf236ce13014574f002869b80e0165d0602083822c" Oct 30 00:30:04.554222 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:04.554208 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb3ad-6f6bb" Oct 30 00:30:04.577085 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:04.576925 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr" podStartSLOduration=4.576897616 podStartE2EDuration="4.576897616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:30:04.533265507 +0000 UTC m=+1519.738766691" watchObservedRunningTime="2026-10-30 00:30:04.576897616 +0000 UTC m=+1519.782398797" Oct 30 00:30:05.566039 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:05.565999 5001 generic.go:356] "Generic (PLEG): container finished" podID="3e472400-6cd4-4ef4-a999-5717b9570164" containerID="dc2e619d07e1539475f62f22a0f43e2b4ae72280bc5922d58425897e91e51513" exitCode=60 Oct 30 00:30:05.566510 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:05.566282 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2" event={"ID":"3e472400-6cd4-4ef4-a999-5717b9570164","Type":"ContainerDied","Data":"dc2e619d07e1539475f62f22a0f43e2b4ae72280bc5922d58425897e91e51513"} Oct 30 00:30:05.566510 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:05.566350 5001 scope.go:122] "RemoveContainer" containerID="85e8aee40f8f295ade83b3b38efc7f506c16125d0a7cb32272208db2a3280422" Oct 30 00:30:05.566600 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:05.566525 5001 scope.go:122] "RemoveContainer" containerID="dc2e619d07e1539475f62f22a0f43e2b4ae72280bc5922d58425897e91e51513" Oct 30 00:30:05.566988 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:30:05.566962 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2_odh-ai-gateway-infra(3e472400-6cd4-4ef4-a999-5717b9570164)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2" podUID="3e472400-6cd4-4ef4-a999-5717b9570164" Oct 30 00:30:05.568860 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:05.568836 5001 generic.go:356] "Generic (PLEG): container finished" podID="dd6aa747-bb02-4dda-8570-745914ae491f" containerID="7a272978608e817ddb69a288ddea927ebb1e5ac7c7585ac0abf1beeffb48f549" exitCode=60 Oct 30 00:30:05.569000 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:05.568952 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt" event={"ID":"dd6aa747-bb02-4dda-8570-745914ae491f","Type":"ContainerDied","Data":"7a272978608e817ddb69a288ddea927ebb1e5ac7c7585ac0abf1beeffb48f549"} Oct 30 00:30:05.569233 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:05.569218 5001 scope.go:122] "RemoveContainer" containerID="7a272978608e817ddb69a288ddea927ebb1e5ac7c7585ac0abf1beeffb48f549" Oct 30 00:30:05.569580 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:30:05.569558 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt_odh-ai-gateway-infra(dd6aa747-bb02-4dda-8570-745914ae491f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt" podUID="dd6aa747-bb02-4dda-8570-745914ae491f" Oct 30 00:30:05.571235 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:05.571212 5001 generic.go:356] "Generic (PLEG): container finished" podID="de5e2310-d7f8-45df-8613-6977a1ef3866" containerID="7aebf6e7724a3d729b8301350aea8e80185b7ba7876fd88619b868f99177c361" exitCode=60 Oct 30 00:30:05.571415 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:05.571289 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6" event={"ID":"de5e2310-d7f8-45df-8613-6977a1ef3866","Type":"ContainerDied","Data":"7aebf6e7724a3d729b8301350aea8e80185b7ba7876fd88619b868f99177c361"} Oct 30 00:30:05.571732 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:05.571549 5001 scope.go:122] "RemoveContainer" containerID="7aebf6e7724a3d729b8301350aea8e80185b7ba7876fd88619b868f99177c361" Oct 30 00:30:05.571856 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:30:05.571838 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6_odh-ai-gateway-infra(de5e2310-d7f8-45df-8613-6977a1ef3866)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6" podUID="de5e2310-d7f8-45df-8613-6977a1ef3866" Oct 30 00:30:05.573885 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:05.573866 5001 generic.go:356] "Generic (PLEG): container finished" podID="2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d" containerID="8a42e4779fabf528e4b89267215511026de2c6979d2c460d97f31c857243d07a" exitCode=60 Oct 30 00:30:05.574122 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:05.573944 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7" event={"ID":"2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d","Type":"ContainerDied","Data":"8a42e4779fabf528e4b89267215511026de2c6979d2c460d97f31c857243d07a"} Oct 30 00:30:05.574521 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:05.574269 5001 scope.go:122] "RemoveContainer" containerID="8a42e4779fabf528e4b89267215511026de2c6979d2c460d97f31c857243d07a" Oct 30 00:30:05.574609 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:30:05.574536 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7_odh-ai-gateway-infra(2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7" podUID="2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d" Oct 30 00:30:05.576367 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:05.576345 5001 generic.go:356] "Generic (PLEG): container finished" podID="4fba3789-18ae-409a-bdf4-8e47e68a46ba" containerID="f7fca1a18cdb18a5098d1e22ad7fdf9c97c52c7ef82bddfd93e6361f14661707" exitCode=60 Oct 30 00:30:05.576436 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:05.576421 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9" event={"ID":"4fba3789-18ae-409a-bdf4-8e47e68a46ba","Type":"ContainerDied","Data":"f7fca1a18cdb18a5098d1e22ad7fdf9c97c52c7ef82bddfd93e6361f14661707"} Oct 30 00:30:05.576644 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:05.576621 5001 scope.go:122] "RemoveContainer" containerID="f7fca1a18cdb18a5098d1e22ad7fdf9c97c52c7ef82bddfd93e6361f14661707" Oct 30 00:30:05.576923 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:30:05.576891 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9_odh-ai-gateway-infra(4fba3789-18ae-409a-bdf4-8e47e68a46ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9" podUID="4fba3789-18ae-409a-bdf4-8e47e68a46ba" Oct 30 00:30:05.580097 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:05.580074 5001 generic.go:356] "Generic (PLEG): container finished" podID="d0d09f85-579f-4726-95a9-b6680576db12" containerID="c5f86f12e7550adccf27510a6946ead0002ed45604d9f736aa3f10210f0a097a" exitCode=60 Oct 30 00:30:05.580193 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:05.580153 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr" event={"ID":"d0d09f85-579f-4726-95a9-b6680576db12","Type":"ContainerDied","Data":"c5f86f12e7550adccf27510a6946ead0002ed45604d9f736aa3f10210f0a097a"} Oct 30 00:30:05.580350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:05.580334 5001 scope.go:122] "RemoveContainer" containerID="c5f86f12e7550adccf27510a6946ead0002ed45604d9f736aa3f10210f0a097a" Oct 30 00:30:05.580575 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:30:05.580556 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr_odh-ai-gateway-infra(d0d09f85-579f-4726-95a9-b6680576db12)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr" podUID="d0d09f85-579f-4726-95a9-b6680576db12" Oct 30 00:30:05.582485 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:05.582463 5001 generic.go:356] "Generic (PLEG): container finished" podID="42f34b56-b044-4973-a43d-6e8e3b8050a9" containerID="d858bc2cfe1f066d105d56817423fea863ebf9d0e295f2210f4990cbc27655e3" exitCode=60 Oct 30 00:30:05.582616 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:05.582487 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf" event={"ID":"42f34b56-b044-4973-a43d-6e8e3b8050a9","Type":"ContainerDied","Data":"d858bc2cfe1f066d105d56817423fea863ebf9d0e295f2210f4990cbc27655e3"} Oct 30 00:30:05.582763 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:05.582746 5001 scope.go:122] "RemoveContainer" containerID="d858bc2cfe1f066d105d56817423fea863ebf9d0e295f2210f4990cbc27655e3" Oct 30 00:30:05.583006 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:30:05.582986 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf_odh-ai-gateway-infra(42f34b56-b044-4973-a43d-6e8e3b8050a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf" podUID="42f34b56-b044-4973-a43d-6e8e3b8050a9" Oct 30 00:30:05.584217 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:05.584209 5001 scope.go:122] "RemoveContainer" containerID="4b26e614d67c35f0120eadebcfb4abea531b2a964409e9d31d41386110c84e59" Oct 30 00:30:05.584556 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:05.584540 5001 generic.go:356] "Generic (PLEG): container finished" podID="2c07d333-2dda-47eb-81e0-4b99d2fffc85" containerID="6c82afc3852f08a68fa18b46d4428e1e8a4f5a7ef0aba8410c1c7dc42611f6e2" exitCode=60 Oct 30 00:30:05.584632 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:05.584565 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888670-279bh" event={"ID":"2c07d333-2dda-47eb-81e0-4b99d2fffc85","Type":"ContainerDied","Data":"6c82afc3852f08a68fa18b46d4428e1e8a4f5a7ef0aba8410c1c7dc42611f6e2"} Oct 30 00:30:05.584781 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:05.584767 5001 scope.go:122] "RemoveContainer" containerID="6c82afc3852f08a68fa18b46d4428e1e8a4f5a7ef0aba8410c1c7dc42611f6e2" Oct 30 00:30:05.585094 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:30:05.584979 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888670-279bh_odh-ai-gateway-infra(2c07d333-2dda-47eb-81e0-4b99d2fffc85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888670-279bh" podUID="2c07d333-2dda-47eb-81e0-4b99d2fffc85" Oct 30 00:30:05.599467 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:05.599450 5001 scope.go:122] "RemoveContainer" containerID="49aed6fa076b5503ddcf409ca5b03f6b795670287e25f57ceba8016272c3d2ec" Oct 30 00:30:05.620156 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:05.620133 5001 scope.go:122] "RemoveContainer" containerID="8700bafb7b23777de0ee42fd3804b1e6c048684cdbd6fe2a884cadcc5e44af44" Oct 30 00:30:05.633889 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:05.633868 5001 scope.go:122] "RemoveContainer" containerID="1b89e4d88536e0260a2bfc04195455eda8e0007b49947ed7cf76bc7dee3cd624" Oct 30 00:30:05.649231 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:05.649209 5001 scope.go:122] "RemoveContainer" containerID="1c354f3c498fa796fbd1c05acb3d5ecf0ca2191d3251dbfd296a14e8daa61b34" Oct 30 00:30:05.663274 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:05.663254 5001 scope.go:122] "RemoveContainer" containerID="8e4f7cce75a36f2a44d1e73c4d2e122eb647f7e763a12f10fb7ec1816b876a55" Oct 30 00:30:05.678405 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:05.678385 5001 scope.go:122] "RemoveContainer" containerID="a4e1c6afcd4a3ad5dea1f770efc6b8c25c84bbd7f7ac6c69f79af6146315a38b" Oct 30 00:30:06.596197 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:06.596166 5001 scope.go:122] "RemoveContainer" containerID="c5f86f12e7550adccf27510a6946ead0002ed45604d9f736aa3f10210f0a097a" Oct 30 00:30:06.596660 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:30:06.596504 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr_odh-ai-gateway-infra(d0d09f85-579f-4726-95a9-b6680576db12)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr" podUID="d0d09f85-579f-4726-95a9-b6680576db12" Oct 30 00:30:06.597983 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:06.597965 5001 scope.go:122] "RemoveContainer" containerID="d858bc2cfe1f066d105d56817423fea863ebf9d0e295f2210f4990cbc27655e3" Oct 30 00:30:06.598218 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:30:06.598197 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf_odh-ai-gateway-infra(42f34b56-b044-4973-a43d-6e8e3b8050a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf" podUID="42f34b56-b044-4973-a43d-6e8e3b8050a9" Oct 30 00:30:06.599570 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:06.599549 5001 scope.go:122] "RemoveContainer" containerID="6c82afc3852f08a68fa18b46d4428e1e8a4f5a7ef0aba8410c1c7dc42611f6e2" Oct 30 00:30:06.599802 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:30:06.599774 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888670-279bh_odh-ai-gateway-infra(2c07d333-2dda-47eb-81e0-4b99d2fffc85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888670-279bh" podUID="2c07d333-2dda-47eb-81e0-4b99d2fffc85" Oct 30 00:30:06.601197 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:06.601180 5001 scope.go:122] "RemoveContainer" containerID="dc2e619d07e1539475f62f22a0f43e2b4ae72280bc5922d58425897e91e51513" Oct 30 00:30:06.601406 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:30:06.601384 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2_odh-ai-gateway-infra(3e472400-6cd4-4ef4-a999-5717b9570164)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2" podUID="3e472400-6cd4-4ef4-a999-5717b9570164" Oct 30 00:30:06.602710 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:06.602695 5001 scope.go:122] "RemoveContainer" containerID="7a272978608e817ddb69a288ddea927ebb1e5ac7c7585ac0abf1beeffb48f549" Oct 30 00:30:06.602897 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:30:06.602879 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt_odh-ai-gateway-infra(dd6aa747-bb02-4dda-8570-745914ae491f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt" podUID="dd6aa747-bb02-4dda-8570-745914ae491f" Oct 30 00:30:06.604420 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:06.604400 5001 scope.go:122] "RemoveContainer" containerID="7aebf6e7724a3d729b8301350aea8e80185b7ba7876fd88619b868f99177c361" Oct 30 00:30:06.604632 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:30:06.604610 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6_odh-ai-gateway-infra(de5e2310-d7f8-45df-8613-6977a1ef3866)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6" podUID="de5e2310-d7f8-45df-8613-6977a1ef3866" Oct 30 00:30:06.606025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:06.606007 5001 scope.go:122] "RemoveContainer" containerID="8a42e4779fabf528e4b89267215511026de2c6979d2c460d97f31c857243d07a" Oct 30 00:30:06.606228 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:30:06.606207 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7_odh-ai-gateway-infra(2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7" podUID="2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d" Oct 30 00:30:06.607558 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:06.607542 5001 scope.go:122] "RemoveContainer" containerID="f7fca1a18cdb18a5098d1e22ad7fdf9c97c52c7ef82bddfd93e6361f14661707" Oct 30 00:30:06.607739 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:30:06.607721 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9_odh-ai-gateway-infra(4fba3789-18ae-409a-bdf4-8e47e68a46ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9" podUID="4fba3789-18ae-409a-bdf4-8e47e68a46ba" Oct 30 00:30:07.027045 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:07.027013 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe1626e5-fh6kd"] Oct 30 00:30:07.027727 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:07.027714 5001 cpu_manager.go:414] "containerMap: removing container" podUID="3d35cbc6-e2c5-43a5-9749-3785f2dfdde1" containerName="revoke-keys" Oct 30 00:30:07.027785 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:07.027729 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d35cbc6-e2c5-43a5-9749-3785f2dfdde1" containerName="revoke-keys" Oct 30 00:30:07.027838 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:07.027829 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d35cbc6-e2c5-43a5-9749-3785f2dfdde1" containerName="revoke-keys" Oct 30 00:30:07.032821 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:07.032803 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1626e5-fh6kd" Oct 30 00:30:07.046678 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:07.046655 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:30:07.052696 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:07.052670 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe1626e5-fh6kd"] Oct 30 00:30:07.104233 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:07.104198 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16378f33-f8e3-4810-b03a-e4e67067d5a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe1626e5-fh6kd\" (UID: \"16378f33-f8e3-4810-b03a-e4e67067d5a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1626e5-fh6kd" Oct 30 00:30:07.104495 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:07.104467 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49d96\" (UniqueName: \"kubernetes.io/projected/16378f33-f8e3-4810-b03a-e4e67067d5a3-kube-api-access-49d96\") pod \"maas-api-revoke-sub-fe1626e5-fh6kd\" (UID: \"16378f33-f8e3-4810-b03a-e4e67067d5a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1626e5-fh6kd" Oct 30 00:30:07.104624 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:07.104516 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16378f33-f8e3-4810-b03a-e4e67067d5a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe1626e5-fh6kd\" (UID: \"16378f33-f8e3-4810-b03a-e4e67067d5a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1626e5-fh6kd" Oct 30 00:30:07.205473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:07.205442 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49d96\" (UniqueName: \"kubernetes.io/projected/16378f33-f8e3-4810-b03a-e4e67067d5a3-kube-api-access-49d96\") pod \"maas-api-revoke-sub-fe1626e5-fh6kd\" (UID: \"16378f33-f8e3-4810-b03a-e4e67067d5a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1626e5-fh6kd" Oct 30 00:30:07.205689 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:07.205500 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16378f33-f8e3-4810-b03a-e4e67067d5a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe1626e5-fh6kd\" (UID: \"16378f33-f8e3-4810-b03a-e4e67067d5a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1626e5-fh6kd" Oct 30 00:30:07.205689 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:07.205593 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16378f33-f8e3-4810-b03a-e4e67067d5a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe1626e5-fh6kd\" (UID: \"16378f33-f8e3-4810-b03a-e4e67067d5a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1626e5-fh6kd" Oct 30 00:30:07.206189 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:07.206167 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16378f33-f8e3-4810-b03a-e4e67067d5a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe1626e5-fh6kd\" (UID: \"16378f33-f8e3-4810-b03a-e4e67067d5a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1626e5-fh6kd" Oct 30 00:30:07.208057 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:07.207966 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16378f33-f8e3-4810-b03a-e4e67067d5a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe1626e5-fh6kd\" (UID: \"16378f33-f8e3-4810-b03a-e4e67067d5a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1626e5-fh6kd" Oct 30 00:30:07.226353 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:07.226330 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49d96\" (UniqueName: \"kubernetes.io/projected/16378f33-f8e3-4810-b03a-e4e67067d5a3-kube-api-access-49d96\") pod \"maas-api-revoke-sub-fe1626e5-fh6kd\" (UID: \"16378f33-f8e3-4810-b03a-e4e67067d5a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1626e5-fh6kd" Oct 30 00:30:07.358747 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:07.358654 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1626e5-fh6kd" Oct 30 00:30:07.725028 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:07.724904 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf"] Oct 30 00:30:07.725783 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:07.725749 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" podUID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerName="istio-proxy" containerID="cri-o://9a852f3a59bf02b2bf7e1dd63e3aedacc9d8dc7e0e6ab0886b09209460f4f5e5" gracePeriod=30 Oct 30 00:30:07.759943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:07.759889 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe1626e5-fh6kd"] Oct 30 00:30:07.760513 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:30:07.760488 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16378f33_f8e3_4810_b03a_e4e67067d5a3.slice/crio-69b95a0d7272f19ed729a61f692a18fadf21d4f7dc02f5ba2ea7e5529ec8fb74 WatchSource:0}: Error finding container 69b95a0d7272f19ed729a61f692a18fadf21d4f7dc02f5ba2ea7e5529ec8fb74: Status 404 returned error can't find the container with id 69b95a0d7272f19ed729a61f692a18fadf21d4f7dc02f5ba2ea7e5529ec8fb74 Oct 30 00:30:08.627114 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:08.627081 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1626e5-fh6kd" event={"ID":"16378f33-f8e3-4810-b03a-e4e67067d5a3","Type":"ContainerStarted","Data":"7a56e41df38b169aae2c7005fc13191c3e49768bbac6a268b7e0be48b871d3d2"} Oct 30 00:30:08.627272 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:08.627129 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1626e5-fh6kd" event={"ID":"16378f33-f8e3-4810-b03a-e4e67067d5a3","Type":"ContainerStarted","Data":"69b95a0d7272f19ed729a61f692a18fadf21d4f7dc02f5ba2ea7e5529ec8fb74"} Oct 30 00:30:08.668784 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:08.668736 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1626e5-fh6kd" podStartSLOduration=1.6687222959999999 podStartE2EDuration="1.668722296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:30:08.662850366 +0000 UTC m=+1523.868351548" watchObservedRunningTime="2026-10-30 00:30:08.668722296 +0000 UTC m=+1523.874223498" Oct 30 00:30:09.637897 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:09.637864 5001 generic.go:356] "Generic (PLEG): container finished" podID="16378f33-f8e3-4810-b03a-e4e67067d5a3" containerID="7a56e41df38b169aae2c7005fc13191c3e49768bbac6a268b7e0be48b871d3d2" exitCode=0 Oct 30 00:30:09.638289 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:09.637948 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1626e5-fh6kd" event={"ID":"16378f33-f8e3-4810-b03a-e4e67067d5a3","Type":"ContainerDied","Data":"7a56e41df38b169aae2c7005fc13191c3e49768bbac6a268b7e0be48b871d3d2"} Oct 30 00:30:09.978287 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:09.978194 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg"] Oct 30 00:30:09.983520 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:09.983487 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:30:09.997191 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:09.997165 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-1dd8c1-openshift-default-dockercfg-vgs4z\"" Oct 30 00:30:10.015104 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.015051 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg"] Oct 30 00:30:10.127937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.124187 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istio-token\") pod \"e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:30:10.127937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.124334 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istio-envoy\") pod \"e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:30:10.127937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.124376 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-credential-socket\") pod \"e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:30:10.127937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.124439 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxwc4\" (UniqueName: \"kubernetes.io/projected/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-kube-api-access-rxwc4\") pod \"e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:30:10.127937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.124552 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-workload-certs\") pod \"e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:30:10.127937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.124612 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-workload-socket\") pod \"e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:30:10.127937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.124718 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istio-podinfo\") pod \"e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:30:10.127937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.124750 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istiod-ca-cert\") pod \"e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:30:10.127937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.124836 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istio-data\") pod \"e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:30:10.249977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.244663 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istio-token\") pod \"e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:30:10.249977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.244799 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istio-envoy\") pod \"e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:30:10.249977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.244835 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-credential-socket\") pod \"e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:30:10.249977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.244898 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxwc4\" (UniqueName: \"kubernetes.io/projected/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-kube-api-access-rxwc4\") pod \"e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:30:10.249977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.245014 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-workload-certs\") pod \"e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:30:10.249977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.245068 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-workload-socket\") pod \"e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:30:10.249977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.245139 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istio-podinfo\") pod \"e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:30:10.249977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.245172 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istiod-ca-cert\") pod \"e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:30:10.249977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.245226 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istio-data\") pod \"e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:30:10.249977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.245513 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-credential-socket\") pod \"e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:30:10.249977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.245540 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istio-data\") pod \"e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:30:10.249977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.246142 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-workload-socket\") pod \"e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:30:10.249977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.246397 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-workload-certs\") pod \"e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:30:10.249977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.247124 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istiod-ca-cert\") pod \"e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:30:10.271937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.256148 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istio-envoy\") pod \"e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:30:10.282936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.275645 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istio-podinfo\") pod \"e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:30:10.320941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.312899 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxwc4\" (UniqueName: \"kubernetes.io/projected/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-kube-api-access-rxwc4\") pod \"e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:30:10.320941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.313409 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istio-token\") pod \"e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:30:10.604234 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.604201 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:30:10.660647 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.660585 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf"] Oct 30 00:30:10.666814 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.666773 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:30:10.858838 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.858073 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:30:10.858838 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.858224 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:30:10.858838 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.858298 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:30:10.858838 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.858356 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:30:10.858838 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.858390 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ccqw\" (UniqueName: \"kubernetes.io/projected/d34b94b7-c280-4d34-a4b3-5ae295c595cb-kube-api-access-7ccqw\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:30:10.858838 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.858422 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:30:10.858838 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.858464 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:30:10.858838 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.858518 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:30:10.858838 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.858571 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:30:10.880135 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.880102 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn"] Oct 30 00:30:10.889331 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.888740 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:30:10.891983 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.891409 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687"] Oct 30 00:30:10.920126 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.916999 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:30:10.931540 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.931510 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf"] Oct 30 00:30:10.931656 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.931563 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn"] Oct 30 00:30:10.931656 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.931593 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687"] Oct 30 00:30:10.959475 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.959444 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:30:10.959621 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.959559 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:30:10.959621 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.959603 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:30:10.959739 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.959632 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:30:10.959739 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.959688 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:30:10.959739 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.959717 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:30:10.959927 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.959756 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:30:10.959927 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.959796 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:30:10.959927 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.959834 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:30:10.959927 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.959872 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:30:10.960473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.959952 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54658\" (UniqueName: \"kubernetes.io/projected/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-kube-api-access-54658\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:30:10.960473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.959990 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:30:10.960473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.960035 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:30:10.960473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.960063 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:30:10.960473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.960094 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:30:10.960473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.960153 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:30:10.960473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.960199 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:30:10.960473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.960260 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:30:10.960473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.960299 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ccqw\" (UniqueName: \"kubernetes.io/projected/d34b94b7-c280-4d34-a4b3-5ae295c595cb-kube-api-access-7ccqw\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:30:10.960473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.960327 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:30:10.960473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.960349 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llrzt\" (UniqueName: \"kubernetes.io/projected/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-kube-api-access-llrzt\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:30:10.960473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.960437 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:30:10.961701 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.960803 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:30:10.961701 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.960947 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:30:10.961701 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.960979 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:30:10.961701 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.961043 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:30:10.961701 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.961080 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:30:10.961701 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.961113 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:30:10.963617 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.963588 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:30:10.963896 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.963880 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:30:10.964035 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.963953 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:30:10.964210 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.964184 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:30:10.969004 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.964369 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:30:10.969890 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.969866 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:30:10.986549 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.986482 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ccqw\" (UniqueName: \"kubernetes.io/projected/d34b94b7-c280-4d34-a4b3-5ae295c595cb-kube-api-access-7ccqw\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:30:10.988814 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:10.988780 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:30:11.055853 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.055823 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1626e5-fh6kd" Oct 30 00:30:11.062219 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.062189 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16378f33-f8e3-4810-b03a-e4e67067d5a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16378f33-f8e3-4810-b03a-e4e67067d5a3-maas-api-xks-ca\") pod \"16378f33-f8e3-4810-b03a-e4e67067d5a3\" (UID: \"16378f33-f8e3-4810-b03a-e4e67067d5a3\") " Oct 30 00:30:11.062360 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.062246 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16378f33-f8e3-4810-b03a-e4e67067d5a3-kube-api-access-49d96\" (UniqueName: \"kubernetes.io/projected/16378f33-f8e3-4810-b03a-e4e67067d5a3-kube-api-access-49d96\") pod \"16378f33-f8e3-4810-b03a-e4e67067d5a3\" (UID: \"16378f33-f8e3-4810-b03a-e4e67067d5a3\") " Oct 30 00:30:11.062360 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.062308 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16378f33-f8e3-4810-b03a-e4e67067d5a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16378f33-f8e3-4810-b03a-e4e67067d5a3-maas-api-openshift-ca\") pod \"16378f33-f8e3-4810-b03a-e4e67067d5a3\" (UID: \"16378f33-f8e3-4810-b03a-e4e67067d5a3\") " Oct 30 00:30:11.062760 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.062728 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16378f33-f8e3-4810-b03a-e4e67067d5a3-maas-api-openshift-ca" pod "16378f33-f8e3-4810-b03a-e4e67067d5a3" (UID: "16378f33-f8e3-4810-b03a-e4e67067d5a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:30:11.062870 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.062807 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:30:11.062870 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.062842 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:30:11.063039 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.062885 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:30:11.063039 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.062954 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:30:11.063039 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.062993 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:30:11.063277 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.063036 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:30:11.063277 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.063086 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54658\" (UniqueName: \"kubernetes.io/projected/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-kube-api-access-54658\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:30:11.063277 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.063214 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:30:11.063277 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.063247 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:30:11.063498 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.063285 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:30:11.063498 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.063370 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:30:11.063498 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.063451 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:30:11.063662 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.063525 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:30:11.063662 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.063598 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-workload-certs\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:30:11.063817 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.063796 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:30:11.063968 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.063942 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llrzt\" (UniqueName: \"kubernetes.io/projected/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-kube-api-access-llrzt\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:30:11.064130 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.064110 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:30:11.064225 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.064215 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:30:11.064293 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.064266 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:30:11.064348 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.064339 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:30:11.064423 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.064371 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:30:11.064503 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.064488 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16378f33-f8e3-4810-b03a-e4e67067d5a3-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:11.064560 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.064526 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:30:11.064816 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.064789 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:30:11.064892 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.064859 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16378f33-f8e3-4810-b03a-e4e67067d5a3-kube-api-access-49d96" pod "16378f33-f8e3-4810-b03a-e4e67067d5a3" (UID: "16378f33-f8e3-4810-b03a-e4e67067d5a3"). InnerVolumeSpecName "kube-api-access-49d96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:30:11.065193 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.065168 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-credential-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:30:11.065433 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.065342 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:30:11.065516 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.065433 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istio-data\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:30:11.065896 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.065869 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-workload-socket\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:30:11.066076 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.066052 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istiod-ca-cert\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:30:11.066453 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.066403 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:30:11.066630 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.066586 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:30:11.066727 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.066658 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16378f33-f8e3-4810-b03a-e4e67067d5a3-maas-api-xks-ca" pod "16378f33-f8e3-4810-b03a-e4e67067d5a3" (UID: "16378f33-f8e3-4810-b03a-e4e67067d5a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:30:11.067257 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.067234 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istio-podinfo\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:30:11.068117 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.068095 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istio-envoy\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:30:11.084983 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.084957 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:30:11.085412 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.085389 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istio-token\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:30:11.085994 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.085968 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54658\" (UniqueName: \"kubernetes.io/projected/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-kube-api-access-54658\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:30:11.087728 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.087702 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llrzt\" (UniqueName: \"kubernetes.io/projected/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-kube-api-access-llrzt\") pod \"e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:30:11.152310 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.152075 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg"] Oct 30 00:30:11.156986 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:30:11.152621 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9aad1e3_a17c_405b_9f00_3d4da7d3ffe2.slice/crio-2ffcd9668c840d886c6372c1a2003e72e36985ec7bc94e28f044ec606957d0ff WatchSource:0}: Error finding container 2ffcd9668c840d886c6372c1a2003e72e36985ec7bc94e28f044ec606957d0ff: Status 404 returned error can't find the container with id 2ffcd9668c840d886c6372c1a2003e72e36985ec7bc94e28f044ec606957d0ff Oct 30 00:30:11.156986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.156256 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:30:11.156986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.156355 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:30:11.156986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.156429 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:30:11.174931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.166717 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16378f33-f8e3-4810-b03a-e4e67067d5a3-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:11.174931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.166769 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49d96\" (UniqueName: \"kubernetes.io/projected/16378f33-f8e3-4810-b03a-e4e67067d5a3-kube-api-access-49d96\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:11.181551 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.181524 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:30:11.213342 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.213185 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:30:11.239249 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.239221 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:30:11.543244 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.543192 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf"] Oct 30 00:30:11.554355 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:30:11.554318 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd34b94b7_c280_4d34_a4b3_5ae295c595cb.slice/crio-a55a8a92ce082cea8f9b043faafda9ced69565c3b10cb1c7a11c3ba674c389aa WatchSource:0}: Error finding container a55a8a92ce082cea8f9b043faafda9ced69565c3b10cb1c7a11c3ba674c389aa: Status 404 returned error can't find the container with id a55a8a92ce082cea8f9b043faafda9ced69565c3b10cb1c7a11c3ba674c389aa Oct 30 00:30:11.558001 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.557676 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:30:11.558001 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.557740 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:30:11.558001 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.557770 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:30:11.624900 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.624867 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn"] Oct 30 00:30:11.640555 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.637669 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:30:11.640555 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.637751 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:30:11.640555 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.637789 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:30:11.641008 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:30:11.640975 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea34fb77_6b08_4939_b1c0_9ba33e8afd0f.slice/crio-167b1993be7149a3ce8fb54ddeb3b980edb4ce7230d83c07711d8d3a765c43f0 WatchSource:0}: Error finding container 167b1993be7149a3ce8fb54ddeb3b980edb4ce7230d83c07711d8d3a765c43f0: Status 404 returned error can't find the container with id 167b1993be7149a3ce8fb54ddeb3b980edb4ce7230d83c07711d8d3a765c43f0 Oct 30 00:30:11.643118 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.643081 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:30:11.643229 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.643157 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:30:11.643229 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.643198 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:30:11.647095 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.647071 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687"] Oct 30 00:30:11.678516 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.678477 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" event={"ID":"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f","Type":"ContainerStarted","Data":"167b1993be7149a3ce8fb54ddeb3b980edb4ce7230d83c07711d8d3a765c43f0"} Oct 30 00:30:11.680762 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.680732 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" event={"ID":"74a414d8-654e-4a80-bfaf-1ea96f68ea8e","Type":"ContainerStarted","Data":"e06d92a194de75b6a754c863aecdf0f098f7cc9a89eb03958462e66dac6b73f6"} Oct 30 00:30:11.682859 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.682837 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1626e5-fh6kd" event={"ID":"16378f33-f8e3-4810-b03a-e4e67067d5a3","Type":"ContainerDied","Data":"69b95a0d7272f19ed729a61f692a18fadf21d4f7dc02f5ba2ea7e5529ec8fb74"} Oct 30 00:30:11.683007 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.682873 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69b95a0d7272f19ed729a61f692a18fadf21d4f7dc02f5ba2ea7e5529ec8fb74" Oct 30 00:30:11.683007 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.682969 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1626e5-fh6kd" Oct 30 00:30:11.686733 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.686708 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" event={"ID":"d34b94b7-c280-4d34-a4b3-5ae295c595cb","Type":"ContainerStarted","Data":"a55a8a92ce082cea8f9b043faafda9ced69565c3b10cb1c7a11c3ba674c389aa"} Oct 30 00:30:11.689959 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.689904 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" event={"ID":"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2","Type":"ContainerStarted","Data":"67bb7ed0f9f134c303e88e150a5b0bb7dbeb03b797af6df909b5320374e7ed95"} Oct 30 00:30:11.690052 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.689983 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" event={"ID":"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2","Type":"ContainerStarted","Data":"2ffcd9668c840d886c6372c1a2003e72e36985ec7bc94e28f044ec606957d0ff"} Oct 30 00:30:11.736681 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.736618 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" podStartSLOduration=1.736595527 podStartE2EDuration="1.736595527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:30:11.728700022 +0000 UTC m=+1526.934201203" watchObservedRunningTime="2026-10-30 00:30:11.736595527 +0000 UTC m=+1526.942096711" Oct 30 00:30:11.790436 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:11.790372 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" podStartSLOduration=2.7903537419999997 podStartE2EDuration="2.790353742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:30:11.783495179 +0000 UTC m=+1526.988996384" watchObservedRunningTime="2026-10-30 00:30:11.790353742 +0000 UTC m=+1526.995854923" Oct 30 00:30:12.185933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.185425 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:30:12.185933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.185584 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 30 00:30:12.185933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.185634 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" podUID="d34b94b7-c280-4d34-a4b3-5ae295c595cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 30 00:30:12.254769 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.252437 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8"] Oct 30 00:30:12.254769 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.253607 5001 cpu_manager.go:414] "containerMap: removing container" podUID="16378f33-f8e3-4810-b03a-e4e67067d5a3" containerName="revoke-keys" Oct 30 00:30:12.254769 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.253629 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16378f33-f8e3-4810-b03a-e4e67067d5a3" containerName="revoke-keys" Oct 30 00:30:12.254769 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.253810 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="16378f33-f8e3-4810-b03a-e4e67067d5a3" containerName="revoke-keys" Oct 30 00:30:12.259675 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.259046 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:30:12.288133 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.288105 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8"] Oct 30 00:30:12.326940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.320620 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:30:12.326940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.320693 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:30:12.326940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.320825 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:30:12.326940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.320858 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:30:12.326940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.320888 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tql2w\" (UniqueName: \"kubernetes.io/projected/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-kube-api-access-tql2w\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:30:12.326940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.320964 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:30:12.326940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.321063 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:30:12.326940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.321112 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:30:12.326940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.321181 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:30:12.444025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.423759 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:30:12.444025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.423886 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:30:12.444025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.423936 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:30:12.444025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.423968 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tql2w\" (UniqueName: \"kubernetes.io/projected/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-kube-api-access-tql2w\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:30:12.444025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.424021 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:30:12.444025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.424100 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:30:12.444025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.424143 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:30:12.444025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.424196 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:30:12.444025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.424283 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:30:12.444025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.425765 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-credential-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:30:12.444025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.432584 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istiod-ca-cert\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:30:12.444025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.432819 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-workload-socket\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:30:12.444025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.434488 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-workload-certs\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:30:12.444025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.434828 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istio-data\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:30:12.470619 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.470587 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istio-podinfo\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:30:12.486286 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.486255 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istio-envoy\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:30:12.487006 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.486968 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tql2w\" (UniqueName: \"kubernetes.io/projected/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-kube-api-access-tql2w\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:30:12.489595 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.489544 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istio-token\") pod \"e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:30:12.597739 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.597624 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:30:12.605459 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.605322 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:30:12.611946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.611881 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:30:12.703980 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.703469 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" event={"ID":"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f","Type":"ContainerStarted","Data":"2424084982f40ddcac1405610a72c7eca9b7f6985742ea3921953c90f8649a98"} Oct 30 00:30:12.710671 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.710217 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" event={"ID":"74a414d8-654e-4a80-bfaf-1ea96f68ea8e","Type":"ContainerStarted","Data":"fbe80941790a78584998bb0942f567da5bb31636353fd30b0cdcf7367a5595a0"} Oct 30 00:30:12.714002 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.713974 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" event={"ID":"d34b94b7-c280-4d34-a4b3-5ae295c595cb","Type":"ContainerStarted","Data":"6b697ac96104253924b98717b1255c234b39e296acada0ae409b63ea60d3159c"} Oct 30 00:30:12.714241 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.714192 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:30:12.717267 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.717248 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:30:12.751121 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.750717 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" podStartSLOduration=2.750703745 podStartE2EDuration="2.750703745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:30:12.746744733 +0000 UTC m=+1527.952245942" watchObservedRunningTime="2026-10-30 00:30:12.750703745 +0000 UTC m=+1527.956204925" Oct 30 00:30:12.848432 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.848374 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" podStartSLOduration=2.848358065 podStartE2EDuration="2.848358065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:30:12.84427517 +0000 UTC m=+1528.049776371" watchObservedRunningTime="2026-10-30 00:30:12.848358065 +0000 UTC m=+1528.053859246" Oct 30 00:30:12.971463 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.971435 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8"] Oct 30 00:30:12.972169 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:30:12.972062 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f93e25c_1a99_48c6_8a27_05e8ea723a6d.slice/crio-b9f415b9ecbc2b5764e7d24cf4d50254ce5d78ae1de15c51dd41effee6602771 WatchSource:0}: Error finding container b9f415b9ecbc2b5764e7d24cf4d50254ce5d78ae1de15c51dd41effee6602771: Status 404 returned error can't find the container with id b9f415b9ecbc2b5764e7d24cf4d50254ce5d78ae1de15c51dd41effee6602771 Oct 30 00:30:12.980407 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.980358 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:30:12.980639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.980602 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:30:12.980848 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:12.980827 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:30:13.060087 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.060062 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:30:13.190025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.189394 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 30 00:30:13.190025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.189471 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" podUID="d34b94b7-c280-4d34-a4b3-5ae295c595cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 30 00:30:13.255988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.245604 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:30:13.299094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.267161 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 00:30:13.299403 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.299371 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" podUID="74a414d8-654e-4a80-bfaf-1ea96f68ea8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 00:30:13.299579 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.299127 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:30:13.299730 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.293813 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-istio-envoy\") pod \"dfe41eb3-2947-447a-aa04-4eca33589f1d\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " Oct 30 00:30:13.300116 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.300091 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-credential-socket\") pod \"dfe41eb3-2947-447a-aa04-4eca33589f1d\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " Oct 30 00:30:13.300393 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.300372 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfe41eb3-2947-447a-aa04-4eca33589f1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe41eb3-2947-447a-aa04-4eca33589f1d-istiod-ca-cert\") pod \"dfe41eb3-2947-447a-aa04-4eca33589f1d\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " Oct 30 00:30:13.300588 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.300556 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-workload-certs\") pod \"dfe41eb3-2947-447a-aa04-4eca33589f1d\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " Oct 30 00:30:13.300946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.300925 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-workload-socket\") pod \"dfe41eb3-2947-447a-aa04-4eca33589f1d\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " Oct 30 00:30:13.301294 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.301265 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfe41eb3-2947-447a-aa04-4eca33589f1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe41eb3-2947-447a-aa04-4eca33589f1d-istio-podinfo\") pod \"dfe41eb3-2947-447a-aa04-4eca33589f1d\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " Oct 30 00:30:13.301384 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.301334 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-istio-data\") pod \"dfe41eb3-2947-447a-aa04-4eca33589f1d\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " Oct 30 00:30:13.301450 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.301391 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfe41eb3-2947-447a-aa04-4eca33589f1d-istio-token\" (UniqueName: \"kubernetes.io/projected/dfe41eb3-2947-447a-aa04-4eca33589f1d-istio-token\") pod \"dfe41eb3-2947-447a-aa04-4eca33589f1d\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " Oct 30 00:30:13.301502 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.301473 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfe41eb3-2947-447a-aa04-4eca33589f1d-kube-api-access-6dbgg\" (UniqueName: \"kubernetes.io/projected/dfe41eb3-2947-447a-aa04-4eca33589f1d-kube-api-access-6dbgg\") pod \"dfe41eb3-2947-447a-aa04-4eca33589f1d\" (UID: \"dfe41eb3-2947-447a-aa04-4eca33589f1d\") " Oct 30 00:30:13.301592 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.267218 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 00:30:13.301759 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.301731 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" podUID="ea34fb77-6b08-4939-b1c0-9ba33e8afd0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 00:30:13.302485 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.302457 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-credential-socket" pod "dfe41eb3-2947-447a-aa04-4eca33589f1d" (UID: "dfe41eb3-2947-447a-aa04-4eca33589f1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:30:13.306043 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.303780 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-istio-data" pod "dfe41eb3-2947-447a-aa04-4eca33589f1d" (UID: "dfe41eb3-2947-447a-aa04-4eca33589f1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:30:13.306394 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.306288 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-workload-certs" pod "dfe41eb3-2947-447a-aa04-4eca33589f1d" (UID: "dfe41eb3-2947-447a-aa04-4eca33589f1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:30:13.306587 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.306568 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-workload-socket" pod "dfe41eb3-2947-447a-aa04-4eca33589f1d" (UID: "dfe41eb3-2947-447a-aa04-4eca33589f1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:30:13.307112 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.307045 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfe41eb3-2947-447a-aa04-4eca33589f1d-istiod-ca-cert" pod "dfe41eb3-2947-447a-aa04-4eca33589f1d" (UID: "dfe41eb3-2947-447a-aa04-4eca33589f1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:30:13.405400 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.405362 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:13.405615 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.405596 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:13.405745 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.405733 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:13.405843 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.405831 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe41eb3-2947-447a-aa04-4eca33589f1d-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:13.405943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.405932 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:13.422945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.420787 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-istio-envoy" pod "dfe41eb3-2947-447a-aa04-4eca33589f1d" (UID: "dfe41eb3-2947-447a-aa04-4eca33589f1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:30:13.423307 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.423271 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfe41eb3-2947-447a-aa04-4eca33589f1d-kube-api-access-6dbgg" pod "dfe41eb3-2947-447a-aa04-4eca33589f1d" (UID: "dfe41eb3-2947-447a-aa04-4eca33589f1d"). InnerVolumeSpecName "kube-api-access-6dbgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:30:13.524221 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.454176 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfe41eb3-2947-447a-aa04-4eca33589f1d-istio-token" pod "dfe41eb3-2947-447a-aa04-4eca33589f1d" (UID: "dfe41eb3-2947-447a-aa04-4eca33589f1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:30:13.524221 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.523311 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe41eb3-2947-447a-aa04-4eca33589f1d-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:13.524221 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.523347 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe41eb3-2947-447a-aa04-4eca33589f1d-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:13.524221 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.523364 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dbgg\" (UniqueName: \"kubernetes.io/projected/dfe41eb3-2947-447a-aa04-4eca33589f1d-kube-api-access-6dbgg\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:13.630956 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.624122 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfe41eb3-2947-447a-aa04-4eca33589f1d-istio-podinfo" pod "dfe41eb3-2947-447a-aa04-4eca33589f1d" (UID: "dfe41eb3-2947-447a-aa04-4eca33589f1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:30:13.630956 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.625481 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe41eb3-2947-447a-aa04-4eca33589f1d-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:13.848565 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.848473 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" event={"ID":"9f93e25c-1a99-48c6-8a27-05e8ea723a6d","Type":"ContainerStarted","Data":"ce14b911b882bfb0ef52d120c36ffd835e196f7d259dc09b7991c7a79b099001"} Oct 30 00:30:13.849168 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.849141 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" event={"ID":"9f93e25c-1a99-48c6-8a27-05e8ea723a6d","Type":"ContainerStarted","Data":"b9f415b9ecbc2b5764e7d24cf4d50254ce5d78ae1de15c51dd41effee6602771"} Oct 30 00:30:13.862423 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.862389 5001 generic.go:356] "Generic (PLEG): container finished" podID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerID="9a852f3a59bf02b2bf7e1dd63e3aedacc9d8dc7e0e6ab0886b09209460f4f5e5" exitCode=0 Oct 30 00:30:13.863927 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.863886 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" Oct 30 00:30:13.877226 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.875372 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" event={"ID":"dfe41eb3-2947-447a-aa04-4eca33589f1d","Type":"ContainerDied","Data":"9a852f3a59bf02b2bf7e1dd63e3aedacc9d8dc7e0e6ab0886b09209460f4f5e5"} Oct 30 00:30:13.877226 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.875435 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf" event={"ID":"dfe41eb3-2947-447a-aa04-4eca33589f1d","Type":"ContainerDied","Data":"b06df223e7d09ecc405586ad9827e0182406447cb7b34cba192bbcb705d0a10c"} Oct 30 00:30:13.877226 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.875466 5001 scope.go:122] "RemoveContainer" containerID="9a852f3a59bf02b2bf7e1dd63e3aedacc9d8dc7e0e6ab0886b09209460f4f5e5" Oct 30 00:30:13.923862 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.923796 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" podStartSLOduration=1.923752741 podStartE2EDuration="1.923752741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:30:13.914101389 +0000 UTC m=+1529.119602607" watchObservedRunningTime="2026-10-30 00:30:13.923752741 +0000 UTC m=+1529.129253923" Oct 30 00:30:13.996550 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:13.996159 5001 scope.go:122] "RemoveContainer" containerID="1604d7cfa69887eecbbdf6d7f18b6f4d47c680b9ef80fce3e6478b12a59d7423" Oct 30 00:30:14.049254 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:14.047065 5001 scope.go:122] "RemoveContainer" containerID="9a852f3a59bf02b2bf7e1dd63e3aedacc9d8dc7e0e6ab0886b09209460f4f5e5" Oct 30 00:30:14.070928 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:30:14.069393 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a852f3a59bf02b2bf7e1dd63e3aedacc9d8dc7e0e6ab0886b09209460f4f5e5\": container with ID starting with 9a852f3a59bf02b2bf7e1dd63e3aedacc9d8dc7e0e6ab0886b09209460f4f5e5 not found: ID does not exist" containerID="9a852f3a59bf02b2bf7e1dd63e3aedacc9d8dc7e0e6ab0886b09209460f4f5e5" Oct 30 00:30:14.070928 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:14.069460 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a852f3a59bf02b2bf7e1dd63e3aedacc9d8dc7e0e6ab0886b09209460f4f5e5"} err="failed to get container status \"9a852f3a59bf02b2bf7e1dd63e3aedacc9d8dc7e0e6ab0886b09209460f4f5e5\": rpc error: code = NotFound desc = could not find container \"9a852f3a59bf02b2bf7e1dd63e3aedacc9d8dc7e0e6ab0886b09209460f4f5e5\": container with ID starting with 9a852f3a59bf02b2bf7e1dd63e3aedacc9d8dc7e0e6ab0886b09209460f4f5e5 not found: ID does not exist" Oct 30 00:30:14.070928 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:14.069498 5001 scope.go:122] "RemoveContainer" containerID="1604d7cfa69887eecbbdf6d7f18b6f4d47c680b9ef80fce3e6478b12a59d7423" Oct 30 00:30:14.070928 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:30:14.070304 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1604d7cfa69887eecbbdf6d7f18b6f4d47c680b9ef80fce3e6478b12a59d7423\": container with ID starting with 1604d7cfa69887eecbbdf6d7f18b6f4d47c680b9ef80fce3e6478b12a59d7423 not found: ID does not exist" containerID="1604d7cfa69887eecbbdf6d7f18b6f4d47c680b9ef80fce3e6478b12a59d7423" Oct 30 00:30:14.070928 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:14.070382 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1604d7cfa69887eecbbdf6d7f18b6f4d47c680b9ef80fce3e6478b12a59d7423"} err="failed to get container status \"1604d7cfa69887eecbbdf6d7f18b6f4d47c680b9ef80fce3e6478b12a59d7423\": rpc error: code = NotFound desc = could not find container \"1604d7cfa69887eecbbdf6d7f18b6f4d47c680b9ef80fce3e6478b12a59d7423\": container with ID starting with 1604d7cfa69887eecbbdf6d7f18b6f4d47c680b9ef80fce3e6478b12a59d7423 not found: ID does not exist" Oct 30 00:30:14.187063 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:14.186971 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf"] Oct 30 00:30:14.193061 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:14.193029 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 30 00:30:14.193352 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:14.193325 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" podUID="d34b94b7-c280-4d34-a4b3-5ae295c595cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 30 00:30:14.200576 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:14.200543 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-43dc72-openshift-default-5fd4584c49-gxcmf"] Oct 30 00:30:14.266671 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:14.266070 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 00:30:14.266671 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:14.266133 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" podUID="ea34fb77-6b08-4939-b1c0-9ba33e8afd0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 00:30:14.267119 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:14.267097 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 00:30:14.267314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:14.267290 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" podUID="74a414d8-654e-4a80-bfaf-1ea96f68ea8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 00:30:14.598130 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:14.598090 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:30:14.600470 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:14.600331 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 00:30:14.600470 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:14.600388 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" podUID="9f93e25c-1a99-48c6-8a27-05e8ea723a6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 00:30:15.005196 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:15.003570 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfe41eb3-2947-447a-aa04-4eca33589f1d" path="/var/lib/kubelet/pods/dfe41eb3-2947-447a-aa04-4eca33589f1d/volumes" Oct 30 00:30:15.187938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:15.183650 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 30 00:30:15.187938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:15.183709 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" podUID="d34b94b7-c280-4d34-a4b3-5ae295c595cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 30 00:30:15.222224 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:15.222191 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 00:30:15.222464 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:15.222440 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" podUID="74a414d8-654e-4a80-bfaf-1ea96f68ea8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 00:30:15.254060 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:15.254018 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 00:30:15.254369 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:15.254339 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" podUID="ea34fb77-6b08-4939-b1c0-9ba33e8afd0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 00:30:15.628936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:15.619727 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 00:30:15.630148 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:15.630112 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" podUID="9f93e25c-1a99-48c6-8a27-05e8ea723a6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 00:30:15.709505 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:15.709447 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:15.801353 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:15.801296 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:15.992996 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:15.990310 5001 scope.go:122] "RemoveContainer" containerID="7aebf6e7724a3d729b8301350aea8e80185b7ba7876fd88619b868f99177c361" Oct 30 00:30:15.992996 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:15.991701 5001 scope.go:122] "RemoveContainer" containerID="dc2e619d07e1539475f62f22a0f43e2b4ae72280bc5922d58425897e91e51513" Oct 30 00:30:16.053954 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:16.043999 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:16.196042 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:16.186980 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 30 00:30:16.196042 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:16.187039 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" podUID="d34b94b7-c280-4d34-a4b3-5ae295c595cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 30 00:30:16.237628 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:16.237591 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 00:30:16.237863 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:16.237837 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" podUID="74a414d8-654e-4a80-bfaf-1ea96f68ea8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 00:30:16.305109 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:16.305074 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 00:30:16.305371 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:16.305342 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" podUID="ea34fb77-6b08-4939-b1c0-9ba33e8afd0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 00:30:16.449988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:16.449952 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1dd8c1-755bcc85b6-2zdss"] Oct 30 00:30:16.509938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:16.490485 5001 cpu_manager.go:414] "containerMap: removing container" podUID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerName="istio-proxy" Oct 30 00:30:16.509938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:16.490518 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerName="istio-proxy" Oct 30 00:30:16.509938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:16.490565 5001 cpu_manager.go:414] "containerMap: removing container" podUID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerName="istio-proxy" Oct 30 00:30:16.509938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:16.490575 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerName="istio-proxy" Oct 30 00:30:16.509938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:16.490826 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerName="istio-proxy" Oct 30 00:30:16.509938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:16.490842 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfe41eb3-2947-447a-aa04-4eca33589f1d" containerName="istio-proxy" Oct 30 00:30:16.729323 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:16.727987 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 00:30:16.729323 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:16.728044 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" podUID="9f93e25c-1a99-48c6-8a27-05e8ea723a6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 00:30:16.738548 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:16.738509 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1dd8c1-755bcc85b6-2zdss" Oct 30 00:30:17.055936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:17.050353 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 30 00:30:17.060999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:16.769003 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:17.071972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:16.742341 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1dd8c1-755bcc85b6-2zdss"] Oct 30 00:30:17.100951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:17.096988 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:17.100951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:16.884005 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65938fbc-9537-4aa2-b0b0-a77ff94384ca-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-1dd8c1-755bcc85b6-2zdss\" (UID: \"65938fbc-9537-4aa2-b0b0-a77ff94384ca\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1dd8c1-755bcc85b6-2zdss" Oct 30 00:30:17.100951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:17.097632 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7h9d\" (UniqueName: \"kubernetes.io/projected/65938fbc-9537-4aa2-b0b0-a77ff94384ca-kube-api-access-h7h9d\") pod \"payload-processing-e2e-praxis-a-1dd8c1-755bcc85b6-2zdss\" (UID: \"65938fbc-9537-4aa2-b0b0-a77ff94384ca\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1dd8c1-755bcc85b6-2zdss" Oct 30 00:30:17.100951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:17.098977 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:17.165949 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:17.145650 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-1dd8c1-dockercfg-hdxqs\"" Oct 30 00:30:17.165949 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:17.149794 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:17.192056 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:17.192011 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:30:17.271998 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:17.205029 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:17.273741 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:17.237057 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-1dd8c1\"" Oct 30 00:30:17.327993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:17.296200 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 30 00:30:17.356935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:17.351969 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" podUID="d34b94b7-c280-4d34-a4b3-5ae295c595cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 30 00:30:17.426056 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:17.426011 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65938fbc-9537-4aa2-b0b0-a77ff94384ca-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-1dd8c1-755bcc85b6-2zdss\" (UID: \"65938fbc-9537-4aa2-b0b0-a77ff94384ca\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1dd8c1-755bcc85b6-2zdss" Oct 30 00:30:17.426420 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:17.426396 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7h9d\" (UniqueName: \"kubernetes.io/projected/65938fbc-9537-4aa2-b0b0-a77ff94384ca-kube-api-access-h7h9d\") pod \"payload-processing-e2e-praxis-a-1dd8c1-755bcc85b6-2zdss\" (UID: \"65938fbc-9537-4aa2-b0b0-a77ff94384ca\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1dd8c1-755bcc85b6-2zdss" Oct 30 00:30:17.428547 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:17.428521 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65938fbc-9537-4aa2-b0b0-a77ff94384ca-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-1dd8c1-755bcc85b6-2zdss\" (UID: \"65938fbc-9537-4aa2-b0b0-a77ff94384ca\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1dd8c1-755bcc85b6-2zdss" Oct 30 00:30:17.457964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:17.432372 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 00:30:17.457964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:17.442637 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" podUID="27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": context deadline exceeded" Oct 30 00:30:17.457964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:17.447680 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 00:30:17.457964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:17.447731 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" podUID="74a414d8-654e-4a80-bfaf-1ea96f68ea8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 00:30:17.457964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:17.447815 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 00:30:17.457964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:17.447850 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" podUID="ea34fb77-6b08-4939-b1c0-9ba33e8afd0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 00:30:17.752188 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:17.752141 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 00:30:17.752449 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:17.752408 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" podUID="9f93e25c-1a99-48c6-8a27-05e8ea723a6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 00:30:17.775173 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:17.773983 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7h9d\" (UniqueName: \"kubernetes.io/projected/65938fbc-9537-4aa2-b0b0-a77ff94384ca-kube-api-access-h7h9d\") pod \"payload-processing-e2e-praxis-a-1dd8c1-755bcc85b6-2zdss\" (UID: \"65938fbc-9537-4aa2-b0b0-a77ff94384ca\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1dd8c1-755bcc85b6-2zdss" Oct 30 00:30:17.798227 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:17.798187 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:30:17.798496 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:17.798469 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:30:17.820545 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:17.799572 5001 scope.go:122] "RemoveContainer" containerID="f7fca1a18cdb18a5098d1e22ad7fdf9c97c52c7ef82bddfd93e6361f14661707" Oct 30 00:30:17.821382 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:17.814294 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1dd8c1-755bcc85b6-2zdss" Oct 30 00:30:17.831823 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:17.831800 5001 scope.go:122] "RemoveContainer" containerID="d858bc2cfe1f066d105d56817423fea863ebf9d0e295f2210f4990cbc27655e3" Oct 30 00:30:17.832523 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:17.832502 5001 scope.go:122] "RemoveContainer" containerID="6c82afc3852f08a68fa18b46d4428e1e8a4f5a7ef0aba8410c1c7dc42611f6e2" Oct 30 00:30:17.833056 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:17.816602 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:18.059077 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:18.054149 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:18.176120 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:18.175131 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:30:18.194673 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:18.184343 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:30:18.311971 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:18.278467 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 00:30:18.340060 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:18.323648 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" podUID="ea34fb77-6b08-4939-b1c0-9ba33e8afd0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 00:30:18.340060 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:18.281148 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:30:18.340060 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:18.281029 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:18.349984 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:18.282213 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 00:30:18.350218 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:18.350186 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" podUID="74a414d8-654e-4a80-bfaf-1ea96f68ea8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 00:30:18.350437 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:18.350418 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:30:18.357943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:18.350595 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:30:18.445170 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:18.445111 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" podUID="27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:18.530943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:18.446222 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" Oct 30 00:30:18.536051 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:18.536011 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:18.544254 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:18.544197 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": context deadline exceeded" Oct 30 00:30:18.549002 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:18.548974 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 00:30:18.609999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:18.559548 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 30 00:30:18.638691 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:18.638648 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" podUID="d34b94b7-c280-4d34-a4b3-5ae295c595cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 30 00:30:18.638965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:18.604373 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 00:30:18.639423 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:18.604334 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:30:18.643089 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:18.643066 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:30:18.697938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:18.603357 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 00:30:18.697938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:18.694176 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" podUID="9f93e25c-1a99-48c6-8a27-05e8ea723a6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 00:30:18.878946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:18.791155 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" podUID="ec207720-74f5-4146-93cc-c633e090a5c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.201:8000/ready\": context deadline exceeded" Oct 30 00:30:18.878946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:18.876886 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:18.899838 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:18.899809 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:30:18.964272 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:18.964152 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:19.055299 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:19.025013 5001 scope.go:122] "RemoveContainer" containerID="7a272978608e817ddb69a288ddea927ebb1e5ac7c7585ac0abf1beeffb48f549" Oct 30 00:30:19.055299 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:19.025701 5001 scope.go:122] "RemoveContainer" containerID="8a42e4779fabf528e4b89267215511026de2c6979d2c460d97f31c857243d07a" Oct 30 00:30:19.064296 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:19.064243 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 30 00:30:19.182033 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:19.161600 5001 scope.go:122] "RemoveContainer" containerID="c5f86f12e7550adccf27510a6946ead0002ed45604d9f736aa3f10210f0a097a" Oct 30 00:30:19.210089 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:19.184521 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:19.232529 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:30:19.210308 5001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e472400_6cd4_4ef4_a999_5717b9570164.slice/crio-conmon-c08d9587df82f00b56663e222e1e4340888d9cbd9feeaad6df3d9f4250635a60.scope\": RecentStats: unable to find data in memory cache]" Oct 30 00:30:19.400493 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:19.400455 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 00:30:19.403954 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:19.400980 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" podUID="74a414d8-654e-4a80-bfaf-1ea96f68ea8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 00:30:19.426261 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:19.426223 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 00:30:19.426511 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:19.426480 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" podUID="ea34fb77-6b08-4939-b1c0-9ba33e8afd0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 00:30:19.444976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:19.438017 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:19.540760 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:19.533316 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:30:19.540760 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:19.539000 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:30:19.590848 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:19.551116 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:19.627956 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:19.601992 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:19.627956 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:19.625114 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 30 00:30:19.627956 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:19.625169 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" podUID="d34b94b7-c280-4d34-a4b3-5ae295c595cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 30 00:30:19.645150 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:19.643704 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 00:30:19.645150 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:19.643776 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" podUID="9f93e25c-1a99-48c6-8a27-05e8ea723a6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 00:30:19.676054 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:19.669954 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:19.686718 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:19.686679 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-322d20-865f5cc95-vkvd2" podUID="339eea71-a503-460d-b45e-94561c780aca" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 00:30:19.722157 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:30:19.722041 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:30:19.980634 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:19.918230 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" podUID="ec207720-74f5-4146-93cc-c633e090a5c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.201:8000/ready\": context deadline exceeded" Oct 30 00:30:20.018258 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:20.008938 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:20.025211 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:20.025166 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" podUID="27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:20.069375 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:20.069331 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:20.345937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:20.345836 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 30 00:30:20.345937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:20.345897 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" podUID="d34b94b7-c280-4d34-a4b3-5ae295c595cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 30 00:30:20.388989 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:20.346343 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 00:30:20.388989 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:20.346380 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" podUID="74a414d8-654e-4a80-bfaf-1ea96f68ea8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 00:30:20.388989 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:20.346445 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:30:20.417944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:20.414184 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 00:30:20.417944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:20.414248 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" podUID="ea34fb77-6b08-4939-b1c0-9ba33e8afd0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 00:30:20.571143 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:20.571101 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:20.595559 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:20.586871 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:30:20.692999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:20.586931 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:30:20.692999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:20.684974 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:20.730941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:20.729934 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 00:30:20.730941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:20.730010 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" podUID="9f93e25c-1a99-48c6-8a27-05e8ea723a6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 00:30:20.791847 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:20.791807 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:20.925577 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:20.925524 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" podUID="ec207720-74f5-4146-93cc-c633e090a5c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.201:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:21.090931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:21.029466 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:21.090931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:21.056177 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" podUID="27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": context deadline exceeded" Oct 30 00:30:21.090931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:21.077997 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:21.207294 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:30:21.207200 5001 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.29s" Oct 30 00:30:21.241597 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:21.241571 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" Oct 30 00:30:21.286936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:21.274749 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 00:30:21.286936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:21.274809 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" podUID="ea34fb77-6b08-4939-b1c0-9ba33e8afd0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 00:30:21.286936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:21.275420 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 00:30:21.286936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:21.275457 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" podUID="74a414d8-654e-4a80-bfaf-1ea96f68ea8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 00:30:21.286936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:21.277484 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 30 00:30:21.286936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:21.277524 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" podUID="d34b94b7-c280-4d34-a4b3-5ae295c595cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 30 00:30:21.396088 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:21.396040 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:21.506977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:21.505793 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:30:21.506977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:21.505871 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:30:21.506977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:21.505936 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" Oct 30 00:30:21.506977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:21.505982 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:30:21.513151 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:21.507868 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" Oct 30 00:30:21.586614 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:21.585378 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:30:21.586614 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:21.585532 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:30:21.590415 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:21.589791 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:30:21.590415 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:21.590036 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:30:21.600020 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:21.599072 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 00:30:21.600020 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:21.599138 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" podUID="9f93e25c-1a99-48c6-8a27-05e8ea723a6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 00:30:21.689340 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:21.689308 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1dd8c1-755bcc85b6-2zdss"] Oct 30 00:30:21.748339 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:21.748308 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2" event={"ID":"3e472400-6cd4-4ef4-a999-5717b9570164","Type":"ContainerStarted","Data":"c08d9587df82f00b56663e222e1e4340888d9cbd9feeaad6df3d9f4250635a60"} Oct 30 00:30:21.764330 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:21.754233 5001 scope.go:122] "RemoveContainer" containerID="c08d9587df82f00b56663e222e1e4340888d9cbd9feeaad6df3d9f4250635a60" Oct 30 00:30:21.764330 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:30:21.755067 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2_odh-ai-gateway-infra(3e472400-6cd4-4ef4-a999-5717b9570164)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2" podUID="3e472400-6cd4-4ef4-a999-5717b9570164" Oct 30 00:30:21.826163 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:21.825796 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6" event={"ID":"de5e2310-d7f8-45df-8613-6977a1ef3866","Type":"ContainerStarted","Data":"c4ab6e42c3d48087d76162090e1051cfc3c85c359eeed701b76bd6bdcd3597a7"} Oct 30 00:30:21.827261 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:21.826444 5001 scope.go:122] "RemoveContainer" containerID="c4ab6e42c3d48087d76162090e1051cfc3c85c359eeed701b76bd6bdcd3597a7" Oct 30 00:30:21.827261 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:30:21.827041 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6_odh-ai-gateway-infra(de5e2310-d7f8-45df-8613-6977a1ef3866)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6" podUID="de5e2310-d7f8-45df-8613-6977a1ef3866" Oct 30 00:30:21.950618 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:21.950593 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:30:22.182140 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:22.181857 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 30 00:30:22.182140 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:22.181928 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" podUID="d34b94b7-c280-4d34-a4b3-5ae295c595cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 30 00:30:22.214757 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:22.214598 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 00:30:22.214757 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:22.214677 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" podUID="74a414d8-654e-4a80-bfaf-1ea96f68ea8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 00:30:22.240493 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:22.240457 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 00:30:22.240692 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:22.240519 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" podUID="ea34fb77-6b08-4939-b1c0-9ba33e8afd0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 00:30:22.602050 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:22.601276 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 00:30:22.602050 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:22.601334 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" podUID="9f93e25c-1a99-48c6-8a27-05e8ea723a6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 00:30:23.044256 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:23.030214 5001 generic.go:356] "Generic (PLEG): container finished" podID="3e472400-6cd4-4ef4-a999-5717b9570164" containerID="c08d9587df82f00b56663e222e1e4340888d9cbd9feeaad6df3d9f4250635a60" exitCode=60 Oct 30 00:30:23.050943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:23.050584 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2" event={"ID":"3e472400-6cd4-4ef4-a999-5717b9570164","Type":"ContainerDied","Data":"c08d9587df82f00b56663e222e1e4340888d9cbd9feeaad6df3d9f4250635a60"} Oct 30 00:30:23.050943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:23.050672 5001 scope.go:122] "RemoveContainer" containerID="dc2e619d07e1539475f62f22a0f43e2b4ae72280bc5922d58425897e91e51513" Oct 30 00:30:23.053951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:23.051319 5001 scope.go:122] "RemoveContainer" containerID="c08d9587df82f00b56663e222e1e4340888d9cbd9feeaad6df3d9f4250635a60" Oct 30 00:30:23.053951 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:30:23.051748 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2_odh-ai-gateway-infra(3e472400-6cd4-4ef4-a999-5717b9570164)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2" podUID="3e472400-6cd4-4ef4-a999-5717b9570164" Oct 30 00:30:23.092969 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:23.092896 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt" event={"ID":"dd6aa747-bb02-4dda-8570-745914ae491f","Type":"ContainerStarted","Data":"04cd7558cf2084386b839eb69f7e84222406f7f7ae79e6c2d84c27ed6ae29cf1"} Oct 30 00:30:23.213981 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:23.190520 5001 generic.go:356] "Generic (PLEG): container finished" podID="de5e2310-d7f8-45df-8613-6977a1ef3866" containerID="c4ab6e42c3d48087d76162090e1051cfc3c85c359eeed701b76bd6bdcd3597a7" exitCode=60 Oct 30 00:30:23.213981 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:23.190711 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6" event={"ID":"de5e2310-d7f8-45df-8613-6977a1ef3866","Type":"ContainerDied","Data":"c4ab6e42c3d48087d76162090e1051cfc3c85c359eeed701b76bd6bdcd3597a7"} Oct 30 00:30:23.213981 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:23.191272 5001 scope.go:122] "RemoveContainer" containerID="c4ab6e42c3d48087d76162090e1051cfc3c85c359eeed701b76bd6bdcd3597a7" Oct 30 00:30:23.213981 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:30:23.191718 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6_odh-ai-gateway-infra(de5e2310-d7f8-45df-8613-6977a1ef3866)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6" podUID="de5e2310-d7f8-45df-8613-6977a1ef3866" Oct 30 00:30:23.258332 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:23.256804 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7" event={"ID":"2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d","Type":"ContainerStarted","Data":"32546a56f66531ad7682027b76f095a51fc687f3819eb43f49589f96f8bc6be3"} Oct 30 00:30:23.310985 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:23.307584 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9" event={"ID":"4fba3789-18ae-409a-bdf4-8e47e68a46ba","Type":"ContainerStarted","Data":"3be22600181f4a1ad76d2d085b624834ee1a936a776731ecc12af1476f33c304"} Oct 30 00:30:23.346779 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:23.334939 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1dd8c1-755bcc85b6-2zdss" event={"ID":"65938fbc-9537-4aa2-b0b0-a77ff94384ca","Type":"ContainerStarted","Data":"dddb0d1ab2dea99b9dc6eb5563c4f90eaa51c66c53f8f715b87feac27cc5d004"} Oct 30 00:30:23.346779 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:23.335011 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1dd8c1-755bcc85b6-2zdss" event={"ID":"65938fbc-9537-4aa2-b0b0-a77ff94384ca","Type":"ContainerStarted","Data":"914c6948740952c3b5c1453aabcf57a56a28b5e1b300cbd3a791aa1f57e7f6e1"} Oct 30 00:30:23.346779 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:23.346024 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-1dd8c1-755bcc85b6-2zdss" Oct 30 00:30:23.363431 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:23.351374 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr" event={"ID":"d0d09f85-579f-4726-95a9-b6680576db12","Type":"ContainerStarted","Data":"cc17eac95ff35638bd08743befb231b300b7f5cf7dc74ee95bbf07104fdbfdb9"} Oct 30 00:30:23.373007 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:23.368872 5001 generic.go:356] "Generic (PLEG): container finished" podID="42f34b56-b044-4973-a43d-6e8e3b8050a9" containerID="a76dd624f58f6bfbc43675b4a4cda7e5bebd1642d54de343a0f52992ad090180" exitCode=60 Oct 30 00:30:23.375227 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:23.373220 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf" event={"ID":"42f34b56-b044-4973-a43d-6e8e3b8050a9","Type":"ContainerDied","Data":"a76dd624f58f6bfbc43675b4a4cda7e5bebd1642d54de343a0f52992ad090180"} Oct 30 00:30:23.375227 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:23.373731 5001 scope.go:122] "RemoveContainer" containerID="a76dd624f58f6bfbc43675b4a4cda7e5bebd1642d54de343a0f52992ad090180" Oct 30 00:30:23.375227 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:30:23.374272 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf_odh-ai-gateway-infra(42f34b56-b044-4973-a43d-6e8e3b8050a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf" podUID="42f34b56-b044-4973-a43d-6e8e3b8050a9" Oct 30 00:30:23.385602 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:23.383009 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:23.407504 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:23.407470 5001 generic.go:356] "Generic (PLEG): container finished" podID="2c07d333-2dda-47eb-81e0-4b99d2fffc85" containerID="6cf8ce4ae47496b9b017a5164210a9082fa4888d27d94785e33756abacf7f0f2" exitCode=60 Oct 30 00:30:23.408975 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:23.408948 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888670-279bh" event={"ID":"2c07d333-2dda-47eb-81e0-4b99d2fffc85","Type":"ContainerDied","Data":"6cf8ce4ae47496b9b017a5164210a9082fa4888d27d94785e33756abacf7f0f2"} Oct 30 00:30:23.409630 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:23.409612 5001 scope.go:122] "RemoveContainer" containerID="6cf8ce4ae47496b9b017a5164210a9082fa4888d27d94785e33756abacf7f0f2" Oct 30 00:30:23.411154 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:30:23.411130 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888670-279bh_odh-ai-gateway-infra(2c07d333-2dda-47eb-81e0-4b99d2fffc85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888670-279bh" podUID="2c07d333-2dda-47eb-81e0-4b99d2fffc85" Oct 30 00:30:23.419043 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:23.419020 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6"] Oct 30 00:30:23.471708 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:23.471587 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2"] Oct 30 00:30:23.537957 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:23.531485 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" podUID="ec207720-74f5-4146-93cc-c633e090a5c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.201:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:23.557839 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:23.557036 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:23.563948 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:23.563528 5001 scope.go:122] "RemoveContainer" containerID="7aebf6e7724a3d729b8301350aea8e80185b7ba7876fd88619b868f99177c361" Oct 30 00:30:23.604771 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:23.604312 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 00:30:23.604771 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:23.604396 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" podUID="9f93e25c-1a99-48c6-8a27-05e8ea723a6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 00:30:23.659938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:23.634432 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-1dd8c1-755bcc85b6-2zdss" podStartSLOduration=7.634409685 podStartE2EDuration="7.634409685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:30:23.387444138 +0000 UTC m=+1538.592945320" watchObservedRunningTime="2026-10-30 00:30:23.634409685 +0000 UTC m=+1538.839910869" Oct 30 00:30:23.801105 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:23.801061 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:23.910351 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:23.910320 5001 scope.go:122] "RemoveContainer" containerID="d858bc2cfe1f066d105d56817423fea863ebf9d0e295f2210f4990cbc27655e3" Oct 30 00:30:23.945033 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:23.944991 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:24.009051 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:24.009020 5001 scope.go:122] "RemoveContainer" containerID="6c82afc3852f08a68fa18b46d4428e1e8a4f5a7ef0aba8410c1c7dc42611f6e2" Oct 30 00:30:24.032959 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:24.031984 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:24.097931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:24.086995 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:24.203083 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:24.202994 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:30:24.203473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:24.203413 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" podUID="d34b94b7-c280-4d34-a4b3-5ae295c595cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:24.259583 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:24.259017 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:30:24.259583 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:24.259076 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" podUID="74a414d8-654e-4a80-bfaf-1ea96f68ea8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:24.301078 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:24.301030 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:30:24.301363 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:24.301331 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" podUID="ea34fb77-6b08-4939-b1c0-9ba33e8afd0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:24.351325 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:24.351287 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt"] Oct 30 00:30:24.487652 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:24.487566 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7"] Oct 30 00:30:24.587737 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:24.586071 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:24.587737 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:24.586708 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:30:25.207032 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:25.140954 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 00:30:25.217224 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:25.217179 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:30:25.217396 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:25.217248 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" podUID="d34b94b7-c280-4d34-a4b3-5ae295c595cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:25.276593 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:25.276556 5001 generic.go:356] "Generic (PLEG): container finished" podID="2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d" containerID="32546a56f66531ad7682027b76f095a51fc687f3819eb43f49589f96f8bc6be3" exitCode=60 Oct 30 00:30:25.280687 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:25.280657 5001 generic.go:356] "Generic (PLEG): container finished" podID="4fba3789-18ae-409a-bdf4-8e47e68a46ba" containerID="3be22600181f4a1ad76d2d085b624834ee1a936a776731ecc12af1476f33c304" exitCode=60 Oct 30 00:30:25.291480 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:25.291450 5001 generic.go:356] "Generic (PLEG): container finished" podID="d0d09f85-579f-4726-95a9-b6680576db12" containerID="cc17eac95ff35638bd08743befb231b300b7f5cf7dc74ee95bbf07104fdbfdb9" exitCode=60 Oct 30 00:30:25.309080 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:25.309045 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:30:25.309366 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:25.309335 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" podUID="ea34fb77-6b08-4939-b1c0-9ba33e8afd0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:25.313095 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:25.313057 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:30:25.313250 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:25.313118 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" podUID="74a414d8-654e-4a80-bfaf-1ea96f68ea8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:25.341076 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:25.339621 5001 generic.go:356] "Generic (PLEG): container finished" podID="dd6aa747-bb02-4dda-8570-745914ae491f" containerID="04cd7558cf2084386b839eb69f7e84222406f7f7ae79e6c2d84c27ed6ae29cf1" exitCode=60 Oct 30 00:30:25.380273 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:25.378969 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9"] Oct 30 00:30:25.380273 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:25.379147 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:30:25.380273 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:25.379189 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7" event={"ID":"2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d","Type":"ContainerDied","Data":"32546a56f66531ad7682027b76f095a51fc687f3819eb43f49589f96f8bc6be3"} Oct 30 00:30:25.380273 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:25.379310 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:30:25.380273 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:25.379349 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9" event={"ID":"4fba3789-18ae-409a-bdf4-8e47e68a46ba","Type":"ContainerDied","Data":"3be22600181f4a1ad76d2d085b624834ee1a936a776731ecc12af1476f33c304"} Oct 30 00:30:25.380273 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:25.379379 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr" event={"ID":"d0d09f85-579f-4726-95a9-b6680576db12","Type":"ContainerDied","Data":"cc17eac95ff35638bd08743befb231b300b7f5cf7dc74ee95bbf07104fdbfdb9"} Oct 30 00:30:25.380273 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:25.379423 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt" event={"ID":"dd6aa747-bb02-4dda-8570-745914ae491f","Type":"ContainerDied","Data":"04cd7558cf2084386b839eb69f7e84222406f7f7ae79e6c2d84c27ed6ae29cf1"} Oct 30 00:30:25.388615 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:25.388588 5001 scope.go:122] "RemoveContainer" containerID="8a42e4779fabf528e4b89267215511026de2c6979d2c460d97f31c857243d07a" Oct 30 00:30:25.490046 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:25.489168 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:30:25.490046 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:25.490013 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:30:25.547961 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:25.538594 5001 scope.go:122] "RemoveContainer" containerID="cc17eac95ff35638bd08743befb231b300b7f5cf7dc74ee95bbf07104fdbfdb9" Oct 30 00:30:25.547961 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:30:25.539188 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr_odh-ai-gateway-infra(d0d09f85-579f-4726-95a9-b6680576db12)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr" podUID="d0d09f85-579f-4726-95a9-b6680576db12" Oct 30 00:30:25.559007 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:25.549159 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:30:25.706078 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:25.706036 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:25.933935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:25.933854 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:26.060952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:26.058757 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:30:26.172226 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:26.146549 5001 scope.go:122] "RemoveContainer" containerID="f7fca1a18cdb18a5098d1e22ad7fdf9c97c52c7ef82bddfd93e6361f14661707" Oct 30 00:30:26.173695 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:26.173662 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" podUID="27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": context deadline exceeded" Oct 30 00:30:26.261548 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:26.190006 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr"] Oct 30 00:30:26.271229 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:26.271185 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 30 00:30:26.271566 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:26.271536 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:30:26.403582 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:26.403536 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 00:30:26.425160 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:26.425109 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:30:26.480951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:26.464667 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" podUID="74a414d8-654e-4a80-bfaf-1ea96f68ea8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:26.674414 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:26.670861 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:30:26.765490 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:26.765460 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:30:26.796135 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:26.794723 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:26.803904 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:26.803010 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:26.828523 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:26.828454 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" podUID="ec207720-74f5-4146-93cc-c633e090a5c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.201:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:26.846969 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:26.841424 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:30:26.846969 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:26.841490 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" podUID="ea34fb77-6b08-4939-b1c0-9ba33e8afd0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:26.884952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:26.877545 5001 scope.go:122] "RemoveContainer" containerID="c5f86f12e7550adccf27510a6946ead0002ed45604d9f736aa3f10210f0a097a" Oct 30 00:30:26.933368 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:26.929467 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 00:30:27.039313 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:27.022319 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:27.162562 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:27.162516 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:27.302272 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:27.302217 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:30:27.320980 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:27.317015 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" podUID="27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": context deadline exceeded" Oct 30 00:30:27.566932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:27.566513 5001 scope.go:122] "RemoveContainer" containerID="7a272978608e817ddb69a288ddea927ebb1e5ac7c7585ac0abf1beeffb48f549" Oct 30 00:30:27.651129 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:27.651095 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:30:27.651467 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:27.651449 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-1dd8c1-755bcc85b6-2zdss" Oct 30 00:30:27.651633 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:27.651621 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" Oct 30 00:30:27.809612 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:27.809584 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" Oct 30 00:30:27.810000 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:27.809960 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:30:27.810197 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:27.810163 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" podUID="74a414d8-654e-4a80-bfaf-1ea96f68ea8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:27.811155 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:27.811121 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:27.853933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:27.814822 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:30:27.903733 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:27.903666 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:30:27.904083 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:27.904050 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" podUID="ea34fb77-6b08-4939-b1c0-9ba33e8afd0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:27.905836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:27.819936 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:30:27.906065 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:27.906046 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:30:27.916933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:27.818593 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7" Oct 30 00:30:28.001744 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:27.957152 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:30:28.001744 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:27.957618 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:30:28.022161 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.021829 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d-kube-api-access-mpzfd\" (UniqueName: \"kubernetes.io/projected/2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d-kube-api-access-mpzfd\") pod \"2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d\" (UID: \"2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d\") " Oct 30 00:30:28.027476 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.027189 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d-maas-api-service-ca\") pod \"2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d\" (UID: \"2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d\") " Oct 30 00:30:28.031338 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.029860 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d-maas-api-service-ca" pod "2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d" (UID: "2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:30:28.054239 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.054210 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:30:28.070638 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.070612 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888670-279bh"] Oct 30 00:30:28.072268 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.066372 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 30 00:30:28.074752 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.074727 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d-kube-api-access-mpzfd" pod "2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d" (UID: "2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d"). InnerVolumeSpecName "kube-api-access-mpzfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:30:28.137531 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.135192 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpzfd\" (UniqueName: \"kubernetes.io/projected/2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d-kube-api-access-mpzfd\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:28.137531 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.135223 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d-maas-api-service-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:28.149039 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.148657 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf"] Oct 30 00:30:28.149039 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.148780 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:30:28.217150 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.216710 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:30:28.301419 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.301389 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:30:28.365814 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.365783 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7" event={"ID":"2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d","Type":"ContainerDied","Data":"525c74a0861c2f3c7d00ca7a0fe7d955ed876f0131e2e28092fe59c4ad77d278"} Oct 30 00:30:28.366095 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.366074 5001 scope.go:122] "RemoveContainer" containerID="32546a56f66531ad7682027b76f095a51fc687f3819eb43f49589f96f8bc6be3" Oct 30 00:30:28.366319 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.366305 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7" Oct 30 00:30:28.468683 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.468654 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:30:28.521255 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.521205 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26375c9f-nh57j"] Oct 30 00:30:28.539892 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.532254 5001 cpu_manager.go:414] "containerMap: removing container" podUID="2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d" containerName="cleanup" Oct 30 00:30:28.539892 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.532285 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d" containerName="cleanup" Oct 30 00:30:28.539892 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.532303 5001 cpu_manager.go:414] "containerMap: removing container" podUID="2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d" containerName="cleanup" Oct 30 00:30:28.539892 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.532311 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d" containerName="cleanup" Oct 30 00:30:28.539892 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.539166 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d" containerName="cleanup" Oct 30 00:30:28.539892 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.539195 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d" containerName="cleanup" Oct 30 00:30:28.548048 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.548022 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26375c9f-nh57j" Oct 30 00:30:28.549460 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.549435 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26375c9f-nh57j"] Oct 30 00:30:28.563091 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.563016 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:30:28.602503 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.597245 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:30:28.602503 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.602088 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:30:28.630738 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.630704 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt" Oct 30 00:30:28.633086 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.632791 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9" Oct 30 00:30:28.635116 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.635095 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr" Oct 30 00:30:28.639376 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.639354 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6" Oct 30 00:30:28.641627 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.641608 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2" Oct 30 00:30:28.657414 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.657391 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888670-279bh" Oct 30 00:30:28.664032 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.664002 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bee48652-da0e-4181-93b1-5e88cc6511d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26375c9f-nh57j\" (UID: \"bee48652-da0e-4181-93b1-5e88cc6511d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26375c9f-nh57j" Oct 30 00:30:28.664167 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.664090 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc5sw\" (UniqueName: \"kubernetes.io/projected/bee48652-da0e-4181-93b1-5e88cc6511d4-kube-api-access-pc5sw\") pod \"maas-api-revoke-sub-26375c9f-nh57j\" (UID: \"bee48652-da0e-4181-93b1-5e88cc6511d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26375c9f-nh57j" Oct 30 00:30:28.664443 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.664425 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bee48652-da0e-4181-93b1-5e88cc6511d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26375c9f-nh57j\" (UID: \"bee48652-da0e-4181-93b1-5e88cc6511d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26375c9f-nh57j" Oct 30 00:30:28.768947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.768580 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c07d333-2dda-47eb-81e0-4b99d2fffc85-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c07d333-2dda-47eb-81e0-4b99d2fffc85-maas-api-service-ca\") pod \"2c07d333-2dda-47eb-81e0-4b99d2fffc85\" (UID: \"2c07d333-2dda-47eb-81e0-4b99d2fffc85\") " Oct 30 00:30:28.768947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.768800 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd6aa747-bb02-4dda-8570-745914ae491f-kube-api-access-qbnw5\" (UniqueName: \"kubernetes.io/projected/dd6aa747-bb02-4dda-8570-745914ae491f-kube-api-access-qbnw5\") pod \"dd6aa747-bb02-4dda-8570-745914ae491f\" (UID: \"dd6aa747-bb02-4dda-8570-745914ae491f\") " Oct 30 00:30:28.768947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.768853 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fba3789-18ae-409a-bdf4-8e47e68a46ba-kube-api-access-6nlvt\" (UniqueName: \"kubernetes.io/projected/4fba3789-18ae-409a-bdf4-8e47e68a46ba-kube-api-access-6nlvt\") pod \"4fba3789-18ae-409a-bdf4-8e47e68a46ba\" (UID: \"4fba3789-18ae-409a-bdf4-8e47e68a46ba\") " Oct 30 00:30:28.768947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.768935 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e472400-6cd4-4ef4-a999-5717b9570164-kube-api-access-m2npd\" (UniqueName: \"kubernetes.io/projected/3e472400-6cd4-4ef4-a999-5717b9570164-kube-api-access-m2npd\") pod \"3e472400-6cd4-4ef4-a999-5717b9570164\" (UID: \"3e472400-6cd4-4ef4-a999-5717b9570164\") " Oct 30 00:30:28.769319 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.768973 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c07d333-2dda-47eb-81e0-4b99d2fffc85-kube-api-access-55r8g\" (UniqueName: \"kubernetes.io/projected/2c07d333-2dda-47eb-81e0-4b99d2fffc85-kube-api-access-55r8g\") pod \"2c07d333-2dda-47eb-81e0-4b99d2fffc85\" (UID: \"2c07d333-2dda-47eb-81e0-4b99d2fffc85\") " Oct 30 00:30:28.769319 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.769032 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fba3789-18ae-409a-bdf4-8e47e68a46ba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fba3789-18ae-409a-bdf4-8e47e68a46ba-maas-api-service-ca\") pod \"4fba3789-18ae-409a-bdf4-8e47e68a46ba\" (UID: \"4fba3789-18ae-409a-bdf4-8e47e68a46ba\") " Oct 30 00:30:28.769319 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.769162 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de5e2310-d7f8-45df-8613-6977a1ef3866-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de5e2310-d7f8-45df-8613-6977a1ef3866-maas-api-service-ca\") pod \"de5e2310-d7f8-45df-8613-6977a1ef3866\" (UID: \"de5e2310-d7f8-45df-8613-6977a1ef3866\") " Oct 30 00:30:28.769319 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.769265 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0d09f85-579f-4726-95a9-b6680576db12-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d09f85-579f-4726-95a9-b6680576db12-maas-api-service-ca\") pod \"d0d09f85-579f-4726-95a9-b6680576db12\" (UID: \"d0d09f85-579f-4726-95a9-b6680576db12\") " Oct 30 00:30:28.769319 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.769295 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de5e2310-d7f8-45df-8613-6977a1ef3866-kube-api-access-8lgnz\" (UniqueName: \"kubernetes.io/projected/de5e2310-d7f8-45df-8613-6977a1ef3866-kube-api-access-8lgnz\") pod \"de5e2310-d7f8-45df-8613-6977a1ef3866\" (UID: \"de5e2310-d7f8-45df-8613-6977a1ef3866\") " Oct 30 00:30:28.769607 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.769393 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0d09f85-579f-4726-95a9-b6680576db12-kube-api-access-7cd88\" (UniqueName: \"kubernetes.io/projected/d0d09f85-579f-4726-95a9-b6680576db12-kube-api-access-7cd88\") pod \"d0d09f85-579f-4726-95a9-b6680576db12\" (UID: \"d0d09f85-579f-4726-95a9-b6680576db12\") " Oct 30 00:30:28.769607 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.769425 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e472400-6cd4-4ef4-a999-5717b9570164-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e472400-6cd4-4ef4-a999-5717b9570164-maas-api-service-ca\") pod \"3e472400-6cd4-4ef4-a999-5717b9570164\" (UID: \"3e472400-6cd4-4ef4-a999-5717b9570164\") " Oct 30 00:30:28.769607 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.769509 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd6aa747-bb02-4dda-8570-745914ae491f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6aa747-bb02-4dda-8570-745914ae491f-maas-api-service-ca\") pod \"dd6aa747-bb02-4dda-8570-745914ae491f\" (UID: \"dd6aa747-bb02-4dda-8570-745914ae491f\") " Oct 30 00:30:28.770034 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.770007 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bee48652-da0e-4181-93b1-5e88cc6511d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26375c9f-nh57j\" (UID: \"bee48652-da0e-4181-93b1-5e88cc6511d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26375c9f-nh57j" Oct 30 00:30:28.770142 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.770107 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc5sw\" (UniqueName: \"kubernetes.io/projected/bee48652-da0e-4181-93b1-5e88cc6511d4-kube-api-access-pc5sw\") pod \"maas-api-revoke-sub-26375c9f-nh57j\" (UID: \"bee48652-da0e-4181-93b1-5e88cc6511d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26375c9f-nh57j" Oct 30 00:30:28.770317 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.770301 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bee48652-da0e-4181-93b1-5e88cc6511d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26375c9f-nh57j\" (UID: \"bee48652-da0e-4181-93b1-5e88cc6511d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26375c9f-nh57j" Oct 30 00:30:28.771186 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.771128 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bee48652-da0e-4181-93b1-5e88cc6511d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26375c9f-nh57j\" (UID: \"bee48652-da0e-4181-93b1-5e88cc6511d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26375c9f-nh57j" Oct 30 00:30:28.785395 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.785342 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c07d333-2dda-47eb-81e0-4b99d2fffc85-maas-api-service-ca" pod "2c07d333-2dda-47eb-81e0-4b99d2fffc85" (UID: "2c07d333-2dda-47eb-81e0-4b99d2fffc85"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:30:28.787867 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.786362 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e472400-6cd4-4ef4-a999-5717b9570164-maas-api-service-ca" pod "3e472400-6cd4-4ef4-a999-5717b9570164" (UID: "3e472400-6cd4-4ef4-a999-5717b9570164"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:30:28.788077 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.788042 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0d09f85-579f-4726-95a9-b6680576db12-maas-api-service-ca" pod "d0d09f85-579f-4726-95a9-b6680576db12" (UID: "d0d09f85-579f-4726-95a9-b6680576db12"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:30:28.788329 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.788310 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd6aa747-bb02-4dda-8570-745914ae491f-maas-api-service-ca" pod "dd6aa747-bb02-4dda-8570-745914ae491f" (UID: "dd6aa747-bb02-4dda-8570-745914ae491f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:30:28.789647 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.789625 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de5e2310-d7f8-45df-8613-6977a1ef3866-maas-api-service-ca" pod "de5e2310-d7f8-45df-8613-6977a1ef3866" (UID: "de5e2310-d7f8-45df-8613-6977a1ef3866"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:30:28.790512 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.790022 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fba3789-18ae-409a-bdf4-8e47e68a46ba-maas-api-service-ca" pod "4fba3789-18ae-409a-bdf4-8e47e68a46ba" (UID: "4fba3789-18ae-409a-bdf4-8e47e68a46ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:30:28.840960 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.839405 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c07d333-2dda-47eb-81e0-4b99d2fffc85-kube-api-access-55r8g" pod "2c07d333-2dda-47eb-81e0-4b99d2fffc85" (UID: "2c07d333-2dda-47eb-81e0-4b99d2fffc85"). InnerVolumeSpecName "kube-api-access-55r8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:30:28.840960 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.840176 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bee48652-da0e-4181-93b1-5e88cc6511d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26375c9f-nh57j\" (UID: \"bee48652-da0e-4181-93b1-5e88cc6511d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26375c9f-nh57j" Oct 30 00:30:28.842906 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.841324 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5e2310-d7f8-45df-8613-6977a1ef3866-kube-api-access-8lgnz" pod "de5e2310-d7f8-45df-8613-6977a1ef3866" (UID: "de5e2310-d7f8-45df-8613-6977a1ef3866"). InnerVolumeSpecName "kube-api-access-8lgnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:30:28.843292 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.843229 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc5sw\" (UniqueName: \"kubernetes.io/projected/bee48652-da0e-4181-93b1-5e88cc6511d4-kube-api-access-pc5sw\") pod \"maas-api-revoke-sub-26375c9f-nh57j\" (UID: \"bee48652-da0e-4181-93b1-5e88cc6511d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26375c9f-nh57j" Oct 30 00:30:28.851003 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.847073 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fba3789-18ae-409a-bdf4-8e47e68a46ba-kube-api-access-6nlvt" pod "4fba3789-18ae-409a-bdf4-8e47e68a46ba" (UID: "4fba3789-18ae-409a-bdf4-8e47e68a46ba"). InnerVolumeSpecName "kube-api-access-6nlvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:30:28.851003 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.848584 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d09f85-579f-4726-95a9-b6680576db12-kube-api-access-7cd88" pod "d0d09f85-579f-4726-95a9-b6680576db12" (UID: "d0d09f85-579f-4726-95a9-b6680576db12"). InnerVolumeSpecName "kube-api-access-7cd88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:30:28.857024 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.855092 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd6aa747-bb02-4dda-8570-745914ae491f-kube-api-access-qbnw5" pod "dd6aa747-bb02-4dda-8570-745914ae491f" (UID: "dd6aa747-bb02-4dda-8570-745914ae491f"). InnerVolumeSpecName "kube-api-access-qbnw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:30:28.857024 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.855173 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e472400-6cd4-4ef4-a999-5717b9570164-kube-api-access-m2npd" pod "3e472400-6cd4-4ef4-a999-5717b9570164" (UID: "3e472400-6cd4-4ef4-a999-5717b9570164"). InnerVolumeSpecName "kube-api-access-m2npd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:30:28.876156 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.876107 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbnw5\" (UniqueName: \"kubernetes.io/projected/dd6aa747-bb02-4dda-8570-745914ae491f-kube-api-access-qbnw5\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:28.876387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.876369 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nlvt\" (UniqueName: \"kubernetes.io/projected/4fba3789-18ae-409a-bdf4-8e47e68a46ba-kube-api-access-6nlvt\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:28.876507 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.876492 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2npd\" (UniqueName: \"kubernetes.io/projected/3e472400-6cd4-4ef4-a999-5717b9570164-kube-api-access-m2npd\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:28.876643 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.876621 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55r8g\" (UniqueName: \"kubernetes.io/projected/2c07d333-2dda-47eb-81e0-4b99d2fffc85-kube-api-access-55r8g\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:28.876767 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.876753 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fba3789-18ae-409a-bdf4-8e47e68a46ba-maas-api-service-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:28.876865 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.876854 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de5e2310-d7f8-45df-8613-6977a1ef3866-maas-api-service-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:28.877023 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.876962 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d09f85-579f-4726-95a9-b6680576db12-maas-api-service-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:28.877023 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.876987 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lgnz\" (UniqueName: \"kubernetes.io/projected/de5e2310-d7f8-45df-8613-6977a1ef3866-kube-api-access-8lgnz\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:28.877023 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.877001 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cd88\" (UniqueName: \"kubernetes.io/projected/d0d09f85-579f-4726-95a9-b6680576db12-kube-api-access-7cd88\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:28.877459 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.877238 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e472400-6cd4-4ef4-a999-5717b9570164-maas-api-service-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:28.877459 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.877262 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6aa747-bb02-4dda-8570-745914ae491f-maas-api-service-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:28.877459 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.877281 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c07d333-2dda-47eb-81e0-4b99d2fffc85-maas-api-service-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:28.921101 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.921074 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf" Oct 30 00:30:28.953223 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.953189 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7"] Oct 30 00:30:28.958765 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.958743 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d3e13d-29888670-hg4n7"] Oct 30 00:30:28.959063 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.957466 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26375c9f-nh57j" Oct 30 00:30:28.978351 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.978318 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42f34b56-b044-4973-a43d-6e8e3b8050a9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42f34b56-b044-4973-a43d-6e8e3b8050a9-maas-api-service-ca\") pod \"42f34b56-b044-4973-a43d-6e8e3b8050a9\" (UID: \"42f34b56-b044-4973-a43d-6e8e3b8050a9\") " Oct 30 00:30:28.978654 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.978632 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42f34b56-b044-4973-a43d-6e8e3b8050a9-kube-api-access-qr529\" (UniqueName: \"kubernetes.io/projected/42f34b56-b044-4973-a43d-6e8e3b8050a9-kube-api-access-qr529\") pod \"42f34b56-b044-4973-a43d-6e8e3b8050a9\" (UID: \"42f34b56-b044-4973-a43d-6e8e3b8050a9\") " Oct 30 00:30:28.982977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:28.982124 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42f34b56-b044-4973-a43d-6e8e3b8050a9-maas-api-service-ca" pod "42f34b56-b044-4973-a43d-6e8e3b8050a9" (UID: "42f34b56-b044-4973-a43d-6e8e3b8050a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:30:29.005531 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.001551 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42f34b56-b044-4973-a43d-6e8e3b8050a9-kube-api-access-qr529" pod "42f34b56-b044-4973-a43d-6e8e3b8050a9" (UID: "42f34b56-b044-4973-a43d-6e8e3b8050a9"). InnerVolumeSpecName "kube-api-access-qr529". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:30:29.023980 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.023950 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93016b-p74tv"] Oct 30 00:30:29.024692 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024670 5001 cpu_manager.go:414] "containerMap: removing container" podUID="3e472400-6cd4-4ef4-a999-5717b9570164" containerName="cleanup" Oct 30 00:30:29.024692 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024686 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e472400-6cd4-4ef4-a999-5717b9570164" containerName="cleanup" Oct 30 00:30:29.024692 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024698 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d0d09f85-579f-4726-95a9-b6680576db12" containerName="cleanup" Oct 30 00:30:29.024955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024703 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0d09f85-579f-4726-95a9-b6680576db12" containerName="cleanup" Oct 30 00:30:29.024955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024712 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4fba3789-18ae-409a-bdf4-8e47e68a46ba" containerName="cleanup" Oct 30 00:30:29.024955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024720 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fba3789-18ae-409a-bdf4-8e47e68a46ba" containerName="cleanup" Oct 30 00:30:29.024955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024731 5001 cpu_manager.go:414] "containerMap: removing container" podUID="2c07d333-2dda-47eb-81e0-4b99d2fffc85" containerName="cleanup" Oct 30 00:30:29.024955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024736 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c07d333-2dda-47eb-81e0-4b99d2fffc85" containerName="cleanup" Oct 30 00:30:29.024955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024743 5001 cpu_manager.go:414] "containerMap: removing container" podUID="dd6aa747-bb02-4dda-8570-745914ae491f" containerName="cleanup" Oct 30 00:30:29.024955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024748 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd6aa747-bb02-4dda-8570-745914ae491f" containerName="cleanup" Oct 30 00:30:29.024955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024761 5001 cpu_manager.go:414] "containerMap: removing container" podUID="dd6aa747-bb02-4dda-8570-745914ae491f" containerName="cleanup" Oct 30 00:30:29.024955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024765 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd6aa747-bb02-4dda-8570-745914ae491f" containerName="cleanup" Oct 30 00:30:29.024955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024775 5001 cpu_manager.go:414] "containerMap: removing container" podUID="de5e2310-d7f8-45df-8613-6977a1ef3866" containerName="cleanup" Oct 30 00:30:29.024955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024779 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de5e2310-d7f8-45df-8613-6977a1ef3866" containerName="cleanup" Oct 30 00:30:29.024955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024789 5001 cpu_manager.go:414] "containerMap: removing container" podUID="de5e2310-d7f8-45df-8613-6977a1ef3866" containerName="cleanup" Oct 30 00:30:29.024955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024797 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de5e2310-d7f8-45df-8613-6977a1ef3866" containerName="cleanup" Oct 30 00:30:29.024955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024810 5001 cpu_manager.go:414] "containerMap: removing container" podUID="3e472400-6cd4-4ef4-a999-5717b9570164" containerName="cleanup" Oct 30 00:30:29.024955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024815 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e472400-6cd4-4ef4-a999-5717b9570164" containerName="cleanup" Oct 30 00:30:29.024955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024821 5001 cpu_manager.go:414] "containerMap: removing container" podUID="42f34b56-b044-4973-a43d-6e8e3b8050a9" containerName="cleanup" Oct 30 00:30:29.024955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024826 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42f34b56-b044-4973-a43d-6e8e3b8050a9" containerName="cleanup" Oct 30 00:30:29.024955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024840 5001 cpu_manager.go:414] "containerMap: removing container" podUID="2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d" containerName="cleanup" Oct 30 00:30:29.024955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024844 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d" containerName="cleanup" Oct 30 00:30:29.024955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024854 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d0d09f85-579f-4726-95a9-b6680576db12" containerName="cleanup" Oct 30 00:30:29.024955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024858 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0d09f85-579f-4726-95a9-b6680576db12" containerName="cleanup" Oct 30 00:30:29.024955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024865 5001 cpu_manager.go:414] "containerMap: removing container" podUID="42f34b56-b044-4973-a43d-6e8e3b8050a9" containerName="cleanup" Oct 30 00:30:29.024955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024872 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42f34b56-b044-4973-a43d-6e8e3b8050a9" containerName="cleanup" Oct 30 00:30:29.024955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024884 5001 cpu_manager.go:414] "containerMap: removing container" podUID="de5e2310-d7f8-45df-8613-6977a1ef3866" containerName="cleanup" Oct 30 00:30:29.024955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024888 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de5e2310-d7f8-45df-8613-6977a1ef3866" containerName="cleanup" Oct 30 00:30:29.024955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024897 5001 cpu_manager.go:414] "containerMap: removing container" podUID="3e472400-6cd4-4ef4-a999-5717b9570164" containerName="cleanup" Oct 30 00:30:29.024955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024903 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e472400-6cd4-4ef4-a999-5717b9570164" containerName="cleanup" Oct 30 00:30:29.024955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024927 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4fba3789-18ae-409a-bdf4-8e47e68a46ba" containerName="cleanup" Oct 30 00:30:29.024955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024933 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fba3789-18ae-409a-bdf4-8e47e68a46ba" containerName="cleanup" Oct 30 00:30:29.024955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024942 5001 cpu_manager.go:414] "containerMap: removing container" podUID="42f34b56-b044-4973-a43d-6e8e3b8050a9" containerName="cleanup" Oct 30 00:30:29.024955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024946 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42f34b56-b044-4973-a43d-6e8e3b8050a9" containerName="cleanup" Oct 30 00:30:29.024955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024956 5001 cpu_manager.go:414] "containerMap: removing container" podUID="2c07d333-2dda-47eb-81e0-4b99d2fffc85" containerName="cleanup" Oct 30 00:30:29.024955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.024960 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c07d333-2dda-47eb-81e0-4b99d2fffc85" containerName="cleanup" Oct 30 00:30:29.026709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.025073 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="de5e2310-d7f8-45df-8613-6977a1ef3866" containerName="cleanup" Oct 30 00:30:29.026709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.025085 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="de5e2310-d7f8-45df-8613-6977a1ef3866" containerName="cleanup" Oct 30 00:30:29.026709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.025092 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c07d333-2dda-47eb-81e0-4b99d2fffc85" containerName="cleanup" Oct 30 00:30:29.026709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.025097 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0d09f85-579f-4726-95a9-b6680576db12" containerName="cleanup" Oct 30 00:30:29.026709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.025102 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e472400-6cd4-4ef4-a999-5717b9570164" containerName="cleanup" Oct 30 00:30:29.026709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.025108 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fba3789-18ae-409a-bdf4-8e47e68a46ba" containerName="cleanup" Oct 30 00:30:29.026709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.025113 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="42f34b56-b044-4973-a43d-6e8e3b8050a9" containerName="cleanup" Oct 30 00:30:29.026709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.025119 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d" containerName="cleanup" Oct 30 00:30:29.026709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.025124 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0d09f85-579f-4726-95a9-b6680576db12" containerName="cleanup" Oct 30 00:30:29.026709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.025128 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="42f34b56-b044-4973-a43d-6e8e3b8050a9" containerName="cleanup" Oct 30 00:30:29.026709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.025134 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="de5e2310-d7f8-45df-8613-6977a1ef3866" containerName="cleanup" Oct 30 00:30:29.026709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.025140 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e472400-6cd4-4ef4-a999-5717b9570164" containerName="cleanup" Oct 30 00:30:29.026709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.025147 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fba3789-18ae-409a-bdf4-8e47e68a46ba" containerName="cleanup" Oct 30 00:30:29.026709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.025155 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="42f34b56-b044-4973-a43d-6e8e3b8050a9" containerName="cleanup" Oct 30 00:30:29.026709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.025164 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd6aa747-bb02-4dda-8570-745914ae491f" containerName="cleanup" Oct 30 00:30:29.026709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.025169 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c07d333-2dda-47eb-81e0-4b99d2fffc85" containerName="cleanup" Oct 30 00:30:29.026709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.025174 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e472400-6cd4-4ef4-a999-5717b9570164" containerName="cleanup" Oct 30 00:30:29.026709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.025179 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0d09f85-579f-4726-95a9-b6680576db12" containerName="cleanup" Oct 30 00:30:29.026709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.025188 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd6aa747-bb02-4dda-8570-745914ae491f" containerName="cleanup" Oct 30 00:30:29.031051 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.031029 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93016b-p74tv" Oct 30 00:30:29.035822 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.035794 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93016b-p74tv"] Oct 30 00:30:29.081893 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.081832 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr529\" (UniqueName: \"kubernetes.io/projected/42f34b56-b044-4973-a43d-6e8e3b8050a9-kube-api-access-qr529\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:29.081893 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.081866 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42f34b56-b044-4973-a43d-6e8e3b8050a9-maas-api-service-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:29.183423 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.183387 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f29e9ff-85ad-4f2c-a27d-798cdf352a57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-93016b-p74tv\" (UID: \"2f29e9ff-85ad-4f2c-a27d-798cdf352a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93016b-p74tv" Oct 30 00:30:29.183611 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.183530 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq5wq\" (UniqueName: \"kubernetes.io/projected/2f29e9ff-85ad-4f2c-a27d-798cdf352a57-kube-api-access-vq5wq\") pod \"maas-api-revoke-keys-e2e-praxis-a-93016b-p74tv\" (UID: \"2f29e9ff-85ad-4f2c-a27d-798cdf352a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93016b-p74tv" Oct 30 00:30:29.183611 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.183574 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f29e9ff-85ad-4f2c-a27d-798cdf352a57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-93016b-p74tv\" (UID: \"2f29e9ff-85ad-4f2c-a27d-798cdf352a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93016b-p74tv" Oct 30 00:30:29.285743 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.285711 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f29e9ff-85ad-4f2c-a27d-798cdf352a57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-93016b-p74tv\" (UID: \"2f29e9ff-85ad-4f2c-a27d-798cdf352a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93016b-p74tv" Oct 30 00:30:29.285924 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.285799 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq5wq\" (UniqueName: \"kubernetes.io/projected/2f29e9ff-85ad-4f2c-a27d-798cdf352a57-kube-api-access-vq5wq\") pod \"maas-api-revoke-keys-e2e-praxis-a-93016b-p74tv\" (UID: \"2f29e9ff-85ad-4f2c-a27d-798cdf352a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93016b-p74tv" Oct 30 00:30:29.285924 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.285845 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f29e9ff-85ad-4f2c-a27d-798cdf352a57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-93016b-p74tv\" (UID: \"2f29e9ff-85ad-4f2c-a27d-798cdf352a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93016b-p74tv" Oct 30 00:30:29.286545 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.286508 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f29e9ff-85ad-4f2c-a27d-798cdf352a57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-93016b-p74tv\" (UID: \"2f29e9ff-85ad-4f2c-a27d-798cdf352a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93016b-p74tv" Oct 30 00:30:29.288196 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.288174 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f29e9ff-85ad-4f2c-a27d-798cdf352a57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-93016b-p74tv\" (UID: \"2f29e9ff-85ad-4f2c-a27d-798cdf352a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93016b-p74tv" Oct 30 00:30:29.306169 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.306144 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq5wq\" (UniqueName: \"kubernetes.io/projected/2f29e9ff-85ad-4f2c-a27d-798cdf352a57-kube-api-access-vq5wq\") pod \"maas-api-revoke-keys-e2e-praxis-a-93016b-p74tv\" (UID: \"2f29e9ff-85ad-4f2c-a27d-798cdf352a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93016b-p74tv" Oct 30 00:30:29.328810 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:30:29.328786 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee48652_da0e_4181_93b1_5e88cc6511d4.slice/crio-4a1d4c7328ca6cfb5a44b264135e5ac9899e0db54837447db0d420c2788571a4 WatchSource:0}: Error finding container 4a1d4c7328ca6cfb5a44b264135e5ac9899e0db54837447db0d420c2788571a4: Status 404 returned error can't find the container with id 4a1d4c7328ca6cfb5a44b264135e5ac9899e0db54837447db0d420c2788571a4 Oct 30 00:30:29.329592 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.329568 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26375c9f-nh57j"] Oct 30 00:30:29.373856 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.373835 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93016b-p74tv" Oct 30 00:30:29.449758 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.449721 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt" event={"ID":"dd6aa747-bb02-4dda-8570-745914ae491f","Type":"ContainerDied","Data":"4f62b134bd66c25ad546d398c36e9af240cf39abe6c693c7a0f50d229ae357b7"} Oct 30 00:30:29.449886 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.449795 5001 scope.go:122] "RemoveContainer" containerID="04cd7558cf2084386b839eb69f7e84222406f7f7ae79e6c2d84c27ed6ae29cf1" Oct 30 00:30:29.450020 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.450001 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt" Oct 30 00:30:29.455285 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.455233 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6" Oct 30 00:30:29.455955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.455907 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6" event={"ID":"de5e2310-d7f8-45df-8613-6977a1ef3866","Type":"ContainerDied","Data":"10379fa518783248a70e509f22bce9ff57aae861eefcd500b8aefe390d42344c"} Oct 30 00:30:29.461430 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.461411 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9" Oct 30 00:30:29.461545 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.461428 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9" event={"ID":"4fba3789-18ae-409a-bdf4-8e47e68a46ba","Type":"ContainerDied","Data":"f9df4acbae401f49fb94239724c0bfe2dce61ae17c720c296391f93394b9c3e2"} Oct 30 00:30:29.463386 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.463357 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26375c9f-nh57j" event={"ID":"bee48652-da0e-4181-93b1-5e88cc6511d4","Type":"ContainerStarted","Data":"4a1d4c7328ca6cfb5a44b264135e5ac9899e0db54837447db0d420c2788571a4"} Oct 30 00:30:29.465437 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.465355 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr" Oct 30 00:30:29.465437 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.465384 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr" event={"ID":"d0d09f85-579f-4726-95a9-b6680576db12","Type":"ContainerDied","Data":"e863616a70f983711b8f7ff8b8f59b913b3ddda324819c7ee8a69b95a533953c"} Oct 30 00:30:29.467640 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.467318 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf" Oct 30 00:30:29.467640 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.467318 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf" event={"ID":"42f34b56-b044-4973-a43d-6e8e3b8050a9","Type":"ContainerDied","Data":"41a0f8c18652878a038698e52b3599c2bd91fa760f62ded545f8f2be6c0ef3af"} Oct 30 00:30:29.470556 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.469554 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888670-279bh" Oct 30 00:30:29.470556 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.469582 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888670-279bh" event={"ID":"2c07d333-2dda-47eb-81e0-4b99d2fffc85","Type":"ContainerDied","Data":"be165850175dd8724cba0d18539a31bacf4933db1641f6a7efc9dd84725e4a0c"} Oct 30 00:30:29.473532 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.473512 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2" Oct 30 00:30:29.474097 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.474077 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2" event={"ID":"3e472400-6cd4-4ef4-a999-5717b9570164","Type":"ContainerDied","Data":"8aaeb43dc6279634f5d777cb45ae483cf59d802eba1fa78f15b9f8b65508096e"} Oct 30 00:30:29.475236 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.475205 5001 scope.go:122] "RemoveContainer" containerID="c4ab6e42c3d48087d76162090e1051cfc3c85c359eeed701b76bd6bdcd3597a7" Oct 30 00:30:29.531461 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.511264 5001 scope.go:122] "RemoveContainer" containerID="3be22600181f4a1ad76d2d085b624834ee1a936a776731ecc12af1476f33c304" Oct 30 00:30:29.531461 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.520249 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt"] Oct 30 00:30:29.531461 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.530688 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a7ac6-29888670-fsmmt"] Oct 30 00:30:29.547258 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.547227 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6"] Oct 30 00:30:29.549905 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.549886 5001 scope.go:122] "RemoveContainer" containerID="cc17eac95ff35638bd08743befb231b300b7f5cf7dc74ee95bbf07104fdbfdb9" Oct 30 00:30:29.552655 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.552620 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-564c8f-29888670-gm6b6"] Oct 30 00:30:29.560587 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.560560 5001 scope.go:122] "RemoveContainer" containerID="a76dd624f58f6bfbc43675b4a4cda7e5bebd1642d54de343a0f52992ad090180" Oct 30 00:30:29.571539 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.571329 5001 scope.go:122] "RemoveContainer" containerID="6cf8ce4ae47496b9b017a5164210a9082fa4888d27d94785e33756abacf7f0f2" Oct 30 00:30:29.581552 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.581533 5001 scope.go:122] "RemoveContainer" containerID="c08d9587df82f00b56663e222e1e4340888d9cbd9feeaad6df3d9f4250635a60" Oct 30 00:30:29.588789 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.588766 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr"] Oct 30 00:30:29.595315 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.595295 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7505bd-29888670-dqtjr"] Oct 30 00:30:29.619950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.619853 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888670-279bh"] Oct 30 00:30:29.625089 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.625067 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888670-279bh"] Oct 30 00:30:29.651130 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.651106 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9"] Oct 30 00:30:29.655546 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.655525 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-02665b-29888670-xqqq9"] Oct 30 00:30:29.681045 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:30:29.681018 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f29e9ff_85ad_4f2c_a27d_798cdf352a57.slice/crio-236462b8f93387871e637251240c2d2913b8f91dc941beec6b5b1147beff2fa0 WatchSource:0}: Error finding container 236462b8f93387871e637251240c2d2913b8f91dc941beec6b5b1147beff2fa0: Status 404 returned error can't find the container with id 236462b8f93387871e637251240c2d2913b8f91dc941beec6b5b1147beff2fa0 Oct 30 00:30:29.682099 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.682072 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93016b-p74tv"] Oct 30 00:30:29.688989 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.688965 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf"] Oct 30 00:30:29.692243 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.692220 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fb3ad-29888670-8hkpf"] Oct 30 00:30:29.717408 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.717382 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2"] Oct 30 00:30:29.723299 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:29.723279 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f649b0-29888670-5jkm2"] Oct 30 00:30:30.489607 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:30.489564 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93016b-p74tv" event={"ID":"2f29e9ff-85ad-4f2c-a27d-798cdf352a57","Type":"ContainerStarted","Data":"fad4d0c3a5c2d060e257ad81dae9090f6cf3d0fe724f30903c34b44cbe5c3350"} Oct 30 00:30:30.489797 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:30.489627 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93016b-p74tv" event={"ID":"2f29e9ff-85ad-4f2c-a27d-798cdf352a57","Type":"ContainerStarted","Data":"236462b8f93387871e637251240c2d2913b8f91dc941beec6b5b1147beff2fa0"} Oct 30 00:30:30.491328 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:30.491301 5001 generic.go:356] "Generic (PLEG): container finished" podID="bee48652-da0e-4181-93b1-5e88cc6511d4" containerID="d6f13f34ea1c7cccc53af9f0dfabfe52acb5d4adcdefd744dcba94f5e5057f7d" exitCode=0 Oct 30 00:30:30.491458 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:30.491385 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26375c9f-nh57j" event={"ID":"bee48652-da0e-4181-93b1-5e88cc6511d4","Type":"ContainerDied","Data":"d6f13f34ea1c7cccc53af9f0dfabfe52acb5d4adcdefd744dcba94f5e5057f7d"} Oct 30 00:30:30.528530 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:30.528477 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93016b-p74tv" podStartSLOduration=2.528462768 podStartE2EDuration="2.528462768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:30:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:30:30.524050913 +0000 UTC m=+1545.729552094" watchObservedRunningTime="2026-10-30 00:30:30.528462768 +0000 UTC m=+1545.733963949" Oct 30 00:30:30.921839 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:30.921770 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c07d333-2dda-47eb-81e0-4b99d2fffc85" path="/var/lib/kubelet/pods/2c07d333-2dda-47eb-81e0-4b99d2fffc85/volumes" Oct 30 00:30:30.922304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:30.922287 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d" path="/var/lib/kubelet/pods/2f2e7e7d-b6b3-4fe2-8d85-1354cfcb269d/volumes" Oct 30 00:30:30.922714 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:30.922701 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e472400-6cd4-4ef4-a999-5717b9570164" path="/var/lib/kubelet/pods/3e472400-6cd4-4ef4-a999-5717b9570164/volumes" Oct 30 00:30:30.923444 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:30.923429 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42f34b56-b044-4973-a43d-6e8e3b8050a9" path="/var/lib/kubelet/pods/42f34b56-b044-4973-a43d-6e8e3b8050a9/volumes" Oct 30 00:30:30.923817 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:30.923805 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fba3789-18ae-409a-bdf4-8e47e68a46ba" path="/var/lib/kubelet/pods/4fba3789-18ae-409a-bdf4-8e47e68a46ba/volumes" Oct 30 00:30:30.924213 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:30.924200 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0d09f85-579f-4726-95a9-b6680576db12" path="/var/lib/kubelet/pods/d0d09f85-579f-4726-95a9-b6680576db12/volumes" Oct 30 00:30:30.924866 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:30.924852 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd6aa747-bb02-4dda-8570-745914ae491f" path="/var/lib/kubelet/pods/dd6aa747-bb02-4dda-8570-745914ae491f/volumes" Oct 30 00:30:30.925260 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:30.925246 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de5e2310-d7f8-45df-8613-6977a1ef3866" path="/var/lib/kubelet/pods/de5e2310-d7f8-45df-8613-6977a1ef3866/volumes" Oct 30 00:30:31.505274 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:31.505237 5001 generic.go:356] "Generic (PLEG): container finished" podID="2f29e9ff-85ad-4f2c-a27d-798cdf352a57" containerID="fad4d0c3a5c2d060e257ad81dae9090f6cf3d0fe724f30903c34b44cbe5c3350" exitCode=0 Oct 30 00:30:31.505463 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:31.505330 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93016b-p74tv" event={"ID":"2f29e9ff-85ad-4f2c-a27d-798cdf352a57","Type":"ContainerDied","Data":"fad4d0c3a5c2d060e257ad81dae9090f6cf3d0fe724f30903c34b44cbe5c3350"} Oct 30 00:30:31.762563 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:31.762526 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26375c9f-nh57j" Oct 30 00:30:31.787195 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:31.787169 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bee48652-da0e-4181-93b1-5e88cc6511d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bee48652-da0e-4181-93b1-5e88cc6511d4-maas-api-openshift-ca\") pod \"bee48652-da0e-4181-93b1-5e88cc6511d4\" (UID: \"bee48652-da0e-4181-93b1-5e88cc6511d4\") " Oct 30 00:30:31.787328 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:31.787222 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bee48652-da0e-4181-93b1-5e88cc6511d4-kube-api-access-pc5sw\" (UniqueName: \"kubernetes.io/projected/bee48652-da0e-4181-93b1-5e88cc6511d4-kube-api-access-pc5sw\") pod \"bee48652-da0e-4181-93b1-5e88cc6511d4\" (UID: \"bee48652-da0e-4181-93b1-5e88cc6511d4\") " Oct 30 00:30:31.787492 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:31.787474 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bee48652-da0e-4181-93b1-5e88cc6511d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bee48652-da0e-4181-93b1-5e88cc6511d4-maas-api-xks-ca\") pod \"bee48652-da0e-4181-93b1-5e88cc6511d4\" (UID: \"bee48652-da0e-4181-93b1-5e88cc6511d4\") " Oct 30 00:30:31.787632 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:31.787508 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bee48652-da0e-4181-93b1-5e88cc6511d4-maas-api-openshift-ca" pod "bee48652-da0e-4181-93b1-5e88cc6511d4" (UID: "bee48652-da0e-4181-93b1-5e88cc6511d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:30:31.788089 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:31.788069 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bee48652-da0e-4181-93b1-5e88cc6511d4-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:31.789281 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:31.789258 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee48652-da0e-4181-93b1-5e88cc6511d4-kube-api-access-pc5sw" pod "bee48652-da0e-4181-93b1-5e88cc6511d4" (UID: "bee48652-da0e-4181-93b1-5e88cc6511d4"). InnerVolumeSpecName "kube-api-access-pc5sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:30:31.789370 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:31.789299 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bee48652-da0e-4181-93b1-5e88cc6511d4-maas-api-xks-ca" pod "bee48652-da0e-4181-93b1-5e88cc6511d4" (UID: "bee48652-da0e-4181-93b1-5e88cc6511d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:30:31.889328 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:31.889298 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bee48652-da0e-4181-93b1-5e88cc6511d4-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:31.889328 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:31.889326 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc5sw\" (UniqueName: \"kubernetes.io/projected/bee48652-da0e-4181-93b1-5e88cc6511d4-kube-api-access-pc5sw\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:32.516475 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:32.516449 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26375c9f-nh57j" Oct 30 00:30:32.516899 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:32.516476 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26375c9f-nh57j" event={"ID":"bee48652-da0e-4181-93b1-5e88cc6511d4","Type":"ContainerDied","Data":"4a1d4c7328ca6cfb5a44b264135e5ac9899e0db54837447db0d420c2788571a4"} Oct 30 00:30:32.516899 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:32.516528 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a1d4c7328ca6cfb5a44b264135e5ac9899e0db54837447db0d420c2788571a4" Oct 30 00:30:32.685636 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:32.685604 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93016b-p74tv" Oct 30 00:30:32.706345 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:32.706320 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f29e9ff-85ad-4f2c-a27d-798cdf352a57-kube-api-access-vq5wq\" (UniqueName: \"kubernetes.io/projected/2f29e9ff-85ad-4f2c-a27d-798cdf352a57-kube-api-access-vq5wq\") pod \"2f29e9ff-85ad-4f2c-a27d-798cdf352a57\" (UID: \"2f29e9ff-85ad-4f2c-a27d-798cdf352a57\") " Oct 30 00:30:32.706537 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:32.706518 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f29e9ff-85ad-4f2c-a27d-798cdf352a57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f29e9ff-85ad-4f2c-a27d-798cdf352a57-maas-api-openshift-ca\") pod \"2f29e9ff-85ad-4f2c-a27d-798cdf352a57\" (UID: \"2f29e9ff-85ad-4f2c-a27d-798cdf352a57\") " Oct 30 00:30:32.706631 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:32.706615 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f29e9ff-85ad-4f2c-a27d-798cdf352a57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f29e9ff-85ad-4f2c-a27d-798cdf352a57-maas-api-xks-ca\") pod \"2f29e9ff-85ad-4f2c-a27d-798cdf352a57\" (UID: \"2f29e9ff-85ad-4f2c-a27d-798cdf352a57\") " Oct 30 00:30:32.706948 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:32.706878 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f29e9ff-85ad-4f2c-a27d-798cdf352a57-maas-api-openshift-ca" pod "2f29e9ff-85ad-4f2c-a27d-798cdf352a57" (UID: "2f29e9ff-85ad-4f2c-a27d-798cdf352a57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:30:32.707293 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:32.707275 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f29e9ff-85ad-4f2c-a27d-798cdf352a57-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:32.708482 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:32.708459 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f29e9ff-85ad-4f2c-a27d-798cdf352a57-kube-api-access-vq5wq" pod "2f29e9ff-85ad-4f2c-a27d-798cdf352a57" (UID: "2f29e9ff-85ad-4f2c-a27d-798cdf352a57"). InnerVolumeSpecName "kube-api-access-vq5wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:30:32.708607 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:32.708569 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f29e9ff-85ad-4f2c-a27d-798cdf352a57-maas-api-xks-ca" pod "2f29e9ff-85ad-4f2c-a27d-798cdf352a57" (UID: "2f29e9ff-85ad-4f2c-a27d-798cdf352a57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:30:32.809030 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:32.808943 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq5wq\" (UniqueName: \"kubernetes.io/projected/2f29e9ff-85ad-4f2c-a27d-798cdf352a57-kube-api-access-vq5wq\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:32.809030 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:32.808973 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f29e9ff-85ad-4f2c-a27d-798cdf352a57-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:33.529323 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:33.529294 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93016b-p74tv" Oct 30 00:30:33.529708 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:33.529331 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93016b-p74tv" event={"ID":"2f29e9ff-85ad-4f2c-a27d-798cdf352a57","Type":"ContainerDied","Data":"236462b8f93387871e637251240c2d2913b8f91dc941beec6b5b1147beff2fa0"} Oct 30 00:30:33.529708 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:33.529383 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="236462b8f93387871e637251240c2d2913b8f91dc941beec6b5b1147beff2fa0" Oct 30 00:30:41.821615 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:41.821576 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4895cea6-5nmqx"] Oct 30 00:30:41.822262 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:41.822247 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d0d09f85-579f-4726-95a9-b6680576db12" containerName="cleanup" Oct 30 00:30:41.822262 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:41.822262 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0d09f85-579f-4726-95a9-b6680576db12" containerName="cleanup" Oct 30 00:30:41.822347 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:41.822282 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4fba3789-18ae-409a-bdf4-8e47e68a46ba" containerName="cleanup" Oct 30 00:30:41.822347 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:41.822289 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fba3789-18ae-409a-bdf4-8e47e68a46ba" containerName="cleanup" Oct 30 00:30:41.822347 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:41.822308 5001 cpu_manager.go:414] "containerMap: removing container" podUID="2c07d333-2dda-47eb-81e0-4b99d2fffc85" containerName="cleanup" Oct 30 00:30:41.822347 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:41.822313 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c07d333-2dda-47eb-81e0-4b99d2fffc85" containerName="cleanup" Oct 30 00:30:41.822347 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:41.822323 5001 cpu_manager.go:414] "containerMap: removing container" podUID="bee48652-da0e-4181-93b1-5e88cc6511d4" containerName="revoke-keys" Oct 30 00:30:41.822347 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:41.822328 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bee48652-da0e-4181-93b1-5e88cc6511d4" containerName="revoke-keys" Oct 30 00:30:41.822347 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:41.822340 5001 cpu_manager.go:414] "containerMap: removing container" podUID="dd6aa747-bb02-4dda-8570-745914ae491f" containerName="cleanup" Oct 30 00:30:41.822347 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:41.822344 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd6aa747-bb02-4dda-8570-745914ae491f" containerName="cleanup" Oct 30 00:30:41.822594 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:41.822358 5001 cpu_manager.go:414] "containerMap: removing container" podUID="2f29e9ff-85ad-4f2c-a27d-798cdf352a57" containerName="revoke-keys" Oct 30 00:30:41.822594 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:41.822363 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f29e9ff-85ad-4f2c-a27d-798cdf352a57" containerName="revoke-keys" Oct 30 00:30:41.822594 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:41.822469 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c07d333-2dda-47eb-81e0-4b99d2fffc85" containerName="cleanup" Oct 30 00:30:41.822594 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:41.822479 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="bee48652-da0e-4181-93b1-5e88cc6511d4" containerName="revoke-keys" Oct 30 00:30:41.822594 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:41.822487 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd6aa747-bb02-4dda-8570-745914ae491f" containerName="cleanup" Oct 30 00:30:41.822594 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:41.822495 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f29e9ff-85ad-4f2c-a27d-798cdf352a57" containerName="revoke-keys" Oct 30 00:30:41.822594 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:41.822502 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fba3789-18ae-409a-bdf4-8e47e68a46ba" containerName="cleanup" Oct 30 00:30:41.826311 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:41.826290 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4895cea6-5nmqx" Oct 30 00:30:41.839349 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:41.839325 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:30:41.839477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:41.839440 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:30:41.840418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:41.840395 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:30:41.840513 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:41.840442 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:30:41.845695 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:41.845668 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4895cea6-5nmqx"] Oct 30 00:30:41.939579 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:41.939545 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01383a75-9ce0-4813-ae3b-2ac0fb292132-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4895cea6-5nmqx\" (UID: \"01383a75-9ce0-4813-ae3b-2ac0fb292132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4895cea6-5nmqx" Oct 30 00:30:41.939752 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:41.939675 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01383a75-9ce0-4813-ae3b-2ac0fb292132-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4895cea6-5nmqx\" (UID: \"01383a75-9ce0-4813-ae3b-2ac0fb292132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4895cea6-5nmqx" Oct 30 00:30:41.939752 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:41.939735 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s9mm\" (UniqueName: \"kubernetes.io/projected/01383a75-9ce0-4813-ae3b-2ac0fb292132-kube-api-access-8s9mm\") pod \"maas-api-revoke-sub-4895cea6-5nmqx\" (UID: \"01383a75-9ce0-4813-ae3b-2ac0fb292132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4895cea6-5nmqx" Oct 30 00:30:42.041515 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:42.041482 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s9mm\" (UniqueName: \"kubernetes.io/projected/01383a75-9ce0-4813-ae3b-2ac0fb292132-kube-api-access-8s9mm\") pod \"maas-api-revoke-sub-4895cea6-5nmqx\" (UID: \"01383a75-9ce0-4813-ae3b-2ac0fb292132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4895cea6-5nmqx" Oct 30 00:30:42.041729 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:42.041708 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01383a75-9ce0-4813-ae3b-2ac0fb292132-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4895cea6-5nmqx\" (UID: \"01383a75-9ce0-4813-ae3b-2ac0fb292132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4895cea6-5nmqx" Oct 30 00:30:42.041957 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:42.041930 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01383a75-9ce0-4813-ae3b-2ac0fb292132-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4895cea6-5nmqx\" (UID: \"01383a75-9ce0-4813-ae3b-2ac0fb292132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4895cea6-5nmqx" Oct 30 00:30:42.042649 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:42.042625 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01383a75-9ce0-4813-ae3b-2ac0fb292132-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4895cea6-5nmqx\" (UID: \"01383a75-9ce0-4813-ae3b-2ac0fb292132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4895cea6-5nmqx" Oct 30 00:30:42.044187 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:42.044160 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01383a75-9ce0-4813-ae3b-2ac0fb292132-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4895cea6-5nmqx\" (UID: \"01383a75-9ce0-4813-ae3b-2ac0fb292132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4895cea6-5nmqx" Oct 30 00:30:42.062095 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:42.062074 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s9mm\" (UniqueName: \"kubernetes.io/projected/01383a75-9ce0-4813-ae3b-2ac0fb292132-kube-api-access-8s9mm\") pod \"maas-api-revoke-sub-4895cea6-5nmqx\" (UID: \"01383a75-9ce0-4813-ae3b-2ac0fb292132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4895cea6-5nmqx" Oct 30 00:30:42.155989 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:42.150956 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4895cea6-5nmqx" Oct 30 00:30:42.527553 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:30:42.527521 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01383a75_9ce0_4813_ae3b_2ac0fb292132.slice/crio-7875d33191d65bacc599fe302a53e0b1cf95e05f1cf13b130fd502ade6aa2cce WatchSource:0}: Error finding container 7875d33191d65bacc599fe302a53e0b1cf95e05f1cf13b130fd502ade6aa2cce: Status 404 returned error can't find the container with id 7875d33191d65bacc599fe302a53e0b1cf95e05f1cf13b130fd502ade6aa2cce Oct 30 00:30:42.528628 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:42.528607 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4895cea6-5nmqx"] Oct 30 00:30:42.659005 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:42.658967 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4895cea6-5nmqx" event={"ID":"01383a75-9ce0-4813-ae3b-2ac0fb292132","Type":"ContainerStarted","Data":"7875d33191d65bacc599fe302a53e0b1cf95e05f1cf13b130fd502ade6aa2cce"} Oct 30 00:30:43.875174 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:43.874196 5001 generic.go:356] "Generic (PLEG): container finished" podID="01383a75-9ce0-4813-ae3b-2ac0fb292132" containerID="80d2597e5cbfa1e846d7fba977afd9b7d58fc884f12475458eee81e6c09362e6" exitCode=0 Oct 30 00:30:43.875174 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:43.874482 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4895cea6-5nmqx" event={"ID":"01383a75-9ce0-4813-ae3b-2ac0fb292132","Type":"ContainerDied","Data":"80d2597e5cbfa1e846d7fba977afd9b7d58fc884f12475458eee81e6c09362e6"} Oct 30 00:30:44.013974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:44.013940 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb3ad-6f6bb"] Oct 30 00:30:44.019359 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:44.019329 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb3ad-6f6bb"] Oct 30 00:30:44.922476 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:44.922440 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d35cbc6-e2c5-43a5-9749-3785f2dfdde1" path="/var/lib/kubelet/pods/3d35cbc6-e2c5-43a5-9749-3785f2dfdde1/volumes" Oct 30 00:30:45.082507 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:45.082478 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4895cea6-5nmqx" Oct 30 00:30:45.124771 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:45.124381 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01383a75-9ce0-4813-ae3b-2ac0fb292132-kube-api-access-8s9mm\" (UniqueName: \"kubernetes.io/projected/01383a75-9ce0-4813-ae3b-2ac0fb292132-kube-api-access-8s9mm\") pod \"01383a75-9ce0-4813-ae3b-2ac0fb292132\" (UID: \"01383a75-9ce0-4813-ae3b-2ac0fb292132\") " Oct 30 00:30:45.124771 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:45.124536 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01383a75-9ce0-4813-ae3b-2ac0fb292132-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01383a75-9ce0-4813-ae3b-2ac0fb292132-maas-api-xks-ca\") pod \"01383a75-9ce0-4813-ae3b-2ac0fb292132\" (UID: \"01383a75-9ce0-4813-ae3b-2ac0fb292132\") " Oct 30 00:30:45.124771 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:45.124648 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01383a75-9ce0-4813-ae3b-2ac0fb292132-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01383a75-9ce0-4813-ae3b-2ac0fb292132-maas-api-openshift-ca\") pod \"01383a75-9ce0-4813-ae3b-2ac0fb292132\" (UID: \"01383a75-9ce0-4813-ae3b-2ac0fb292132\") " Oct 30 00:30:45.125700 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:45.125660 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01383a75-9ce0-4813-ae3b-2ac0fb292132-maas-api-openshift-ca" pod "01383a75-9ce0-4813-ae3b-2ac0fb292132" (UID: "01383a75-9ce0-4813-ae3b-2ac0fb292132"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:30:45.127567 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:45.127539 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01383a75-9ce0-4813-ae3b-2ac0fb292132-maas-api-xks-ca" pod "01383a75-9ce0-4813-ae3b-2ac0fb292132" (UID: "01383a75-9ce0-4813-ae3b-2ac0fb292132"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:30:45.128061 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:45.128034 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01383a75-9ce0-4813-ae3b-2ac0fb292132-kube-api-access-8s9mm" pod "01383a75-9ce0-4813-ae3b-2ac0fb292132" (UID: "01383a75-9ce0-4813-ae3b-2ac0fb292132"). InnerVolumeSpecName "kube-api-access-8s9mm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:30:45.226804 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:45.226770 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s9mm\" (UniqueName: \"kubernetes.io/projected/01383a75-9ce0-4813-ae3b-2ac0fb292132-kube-api-access-8s9mm\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:45.226804 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:45.226798 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01383a75-9ce0-4813-ae3b-2ac0fb292132-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:45.226804 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:45.226807 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01383a75-9ce0-4813-ae3b-2ac0fb292132-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:30:45.982358 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:45.981948 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4895cea6-5nmqx" Oct 30 00:30:45.983183 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:45.983131 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4895cea6-5nmqx" event={"ID":"01383a75-9ce0-4813-ae3b-2ac0fb292132","Type":"ContainerDied","Data":"7875d33191d65bacc599fe302a53e0b1cf95e05f1cf13b130fd502ade6aa2cce"} Oct 30 00:30:45.983298 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:45.983241 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7875d33191d65bacc599fe302a53e0b1cf95e05f1cf13b130fd502ade6aa2cce" Oct 30 00:30:47.328154 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:30:47.328109 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:30:49.398129 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:49.398094 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v"] Oct 30 00:30:56.455073 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.454983 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k"] Oct 30 00:30:56.456152 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.456115 5001 cpu_manager.go:414] "containerMap: removing container" podUID="01383a75-9ce0-4813-ae3b-2ac0fb292132" containerName="revoke-keys" Oct 30 00:30:56.456152 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.456141 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01383a75-9ce0-4813-ae3b-2ac0fb292132" containerName="revoke-keys" Oct 30 00:30:56.456386 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.456370 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="01383a75-9ce0-4813-ae3b-2ac0fb292132" containerName="revoke-keys" Oct 30 00:30:56.461876 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.461849 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:30:56.478118 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.477558 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-bc8b7d-openshift-default-dockercfg-ljdnj\"" Oct 30 00:30:56.501815 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.501709 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k"] Oct 30 00:30:56.505198 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.505164 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-workload-socket\") pod \"e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:30:56.505362 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.505231 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istio-envoy\") pod \"e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:30:56.505362 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.505336 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs5fz\" (UniqueName: \"kubernetes.io/projected/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-kube-api-access-rs5fz\") pod \"e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:30:56.505496 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.505476 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istiod-ca-cert\") pod \"e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:30:56.505585 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.505566 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istio-token\") pod \"e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:30:56.505831 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.505806 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-workload-certs\") pod \"e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:30:56.505972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.505952 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istio-podinfo\") pod \"e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:30:56.506134 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.506110 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istio-data\") pod \"e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:30:56.506205 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.506183 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-credential-socket\") pod \"e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:30:56.608459 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.608418 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istiod-ca-cert\") pod \"e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:30:56.608650 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.608531 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istio-token\") pod \"e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:30:56.608650 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.608586 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-workload-certs\") pod \"e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:30:56.608650 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.608637 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istio-podinfo\") pod \"e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:30:56.608839 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.608696 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istio-data\") pod \"e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:30:56.608839 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.608758 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-credential-socket\") pod \"e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:30:56.608972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.608894 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-workload-socket\") pod \"e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:30:56.609043 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.608984 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istio-envoy\") pod \"e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:30:56.609043 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.609016 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs5fz\" (UniqueName: \"kubernetes.io/projected/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-kube-api-access-rs5fz\") pod \"e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:30:56.609159 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.609085 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-workload-certs\") pod \"e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:30:56.609307 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.609282 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istiod-ca-cert\") pod \"e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:30:56.609375 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.609282 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istio-data\") pod \"e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:30:56.609375 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.609320 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-workload-socket\") pod \"e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:30:56.609729 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.609703 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-credential-socket\") pod \"e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:30:56.611686 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.611658 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istio-envoy\") pod \"e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:30:56.611686 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.611682 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istio-podinfo\") pod \"e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:30:56.630356 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.630331 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istio-token\") pod \"e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:30:56.630469 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.630454 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs5fz\" (UniqueName: \"kubernetes.io/projected/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-kube-api-access-rs5fz\") pod \"e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:30:56.799447 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:56.799245 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:30:57.407292 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:57.407256 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k"] Oct 30 00:30:57.407624 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:30:57.407592 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdd71cc9_c2f9_4924_ab67_4b5f03b08f90.slice/crio-ec5a0e2581f5d15017d42ccee2e55240424ea671f5aa89d7174ef7c3aa84ea1e WatchSource:0}: Error finding container ec5a0e2581f5d15017d42ccee2e55240424ea671f5aa89d7174ef7c3aa84ea1e: Status 404 returned error can't find the container with id ec5a0e2581f5d15017d42ccee2e55240424ea671f5aa89d7174ef7c3aa84ea1e Oct 30 00:30:57.411903 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:57.411841 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:30:57.412204 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:57.412164 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:30:57.412442 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:57.412418 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:30:57.668550 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:57.668457 5001 scope.go:122] "RemoveContainer" containerID="f66c1d659908c8cb0f066ea0d1e15c8f1c92e933289d5bb02a290bde64491b2c" Oct 30 00:30:57.684306 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:57.684275 5001 scope.go:122] "RemoveContainer" containerID="a90f65f8608f50128e6b206a0d82dd1089527785e617d4122065befef5172f1b" Oct 30 00:30:58.191296 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:58.191260 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" event={"ID":"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90","Type":"ContainerStarted","Data":"be7a44843a754b3e018a34b63f1c16f6efb066a59e0d101031e550dd7a7808fe"} Oct 30 00:30:58.191486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:58.191320 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" event={"ID":"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90","Type":"ContainerStarted","Data":"ec5a0e2581f5d15017d42ccee2e55240424ea671f5aa89d7174ef7c3aa84ea1e"} Oct 30 00:30:58.241383 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:58.241187 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" podStartSLOduration=2.241122698 podStartE2EDuration="2.241122698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:30:58.239975217 +0000 UTC m=+1573.445476407" watchObservedRunningTime="2026-10-30 00:30:58.241122698 +0000 UTC m=+1573.446623880" Oct 30 00:30:58.801516 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:58.801481 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:30:58.807142 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:58.807117 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:30:59.205083 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:59.204997 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:30:59.205872 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:30:59.205844 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:31:00.152121 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:00.143537 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93016b-p74tv"] Oct 30 00:31:00.215236 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:00.215203 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93016b-p74tv"] Oct 30 00:31:00.920055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:00.920022 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f29e9ff-85ad-4f2c-a27d-798cdf352a57" path="/var/lib/kubelet/pods/2f29e9ff-85ad-4f2c-a27d-798cdf352a57/volumes" Oct 30 00:31:04.420013 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:04.419966 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" podUID="27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e" containerName="main" containerID="cri-o://cfa161992b61f7c996feb50568f39a57791397e8941cf89f5f1dbbdcec126604" gracePeriod=45 Oct 30 00:31:04.742137 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:04.742109 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" Oct 30 00:31:04.845693 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:04.845662 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-tmp-dir\") pod \"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e\" (UID: \"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e\") " Oct 30 00:31:04.845851 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:04.845814 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-model-cache\") pod \"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e\" (UID: \"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e\") " Oct 30 00:31:04.845930 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:04.845897 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-kube-api-access-cdskl\" (UniqueName: \"kubernetes.io/projected/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-kube-api-access-cdskl\") pod \"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e\" (UID: \"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e\") " Oct 30 00:31:04.845983 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:04.845969 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-tmp-dir" pod "27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e" (UID: "27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:04.846020 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:04.845992 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-tls-certs\" (UniqueName: \"kubernetes.io/secret/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-tls-certs\") pod \"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e\" (UID: \"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e\") " Oct 30 00:31:04.846125 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:04.846104 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-model-cache" pod "27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e" (UID: "27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:04.846163 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:04.846155 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-dshm\") pod \"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e\" (UID: \"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e\") " Oct 30 00:31:04.846199 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:04.846174 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-home\" (UniqueName: \"kubernetes.io/empty-dir/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-home\") pod \"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e\" (UID: \"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e\") " Oct 30 00:31:04.846504 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:04.846479 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-home" pod "27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e" (UID: "27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:04.846780 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:04.846763 5001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-home\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:04.846840 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:04.846799 5001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-tmp-dir\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:04.846840 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:04.846811 5001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-model-cache\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:04.848162 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:04.848137 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-tls-certs" pod "27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e" (UID: "27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:31:04.848450 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:04.848432 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-dshm" pod "27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e" (UID: "27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:04.848512 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:04.848493 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-kube-api-access-cdskl" pod "27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e" (UID: "27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e"). InnerVolumeSpecName "kube-api-access-cdskl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:31:04.970074 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:04.968129 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdskl\" (UniqueName: \"kubernetes.io/projected/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-kube-api-access-cdskl\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:04.970074 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:04.968165 5001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-tls-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:04.970074 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:04.968179 5001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e-dshm\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:05.271653 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:05.271623 5001 generic.go:356] "Generic (PLEG): container finished" podID="27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e" containerID="cfa161992b61f7c996feb50568f39a57791397e8941cf89f5f1dbbdcec126604" exitCode=0 Oct 30 00:31:05.271858 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:05.271668 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" Oct 30 00:31:05.271858 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:05.271700 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" event={"ID":"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e","Type":"ContainerDied","Data":"cfa161992b61f7c996feb50568f39a57791397e8941cf89f5f1dbbdcec126604"} Oct 30 00:31:05.271858 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:05.271758 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v" event={"ID":"27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e","Type":"ContainerDied","Data":"e10024d050750583165eb51e835251e7d8b88e013b1a664be6267f9022cd5094"} Oct 30 00:31:05.271858 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:05.271782 5001 scope.go:122] "RemoveContainer" containerID="cfa161992b61f7c996feb50568f39a57791397e8941cf89f5f1dbbdcec126604" Oct 30 00:31:05.283271 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:05.283254 5001 scope.go:122] "RemoveContainer" containerID="cfa161992b61f7c996feb50568f39a57791397e8941cf89f5f1dbbdcec126604" Oct 30 00:31:05.283548 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:31:05.283523 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfa161992b61f7c996feb50568f39a57791397e8941cf89f5f1dbbdcec126604\": container with ID starting with cfa161992b61f7c996feb50568f39a57791397e8941cf89f5f1dbbdcec126604 not found: ID does not exist" containerID="cfa161992b61f7c996feb50568f39a57791397e8941cf89f5f1dbbdcec126604" Oct 30 00:31:05.283615 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:05.283554 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfa161992b61f7c996feb50568f39a57791397e8941cf89f5f1dbbdcec126604"} err="failed to get container status \"cfa161992b61f7c996feb50568f39a57791397e8941cf89f5f1dbbdcec126604\": rpc error: code = NotFound desc = could not find container \"cfa161992b61f7c996feb50568f39a57791397e8941cf89f5f1dbbdcec126604\": container with ID starting with cfa161992b61f7c996feb50568f39a57791397e8941cf89f5f1dbbdcec126604 not found: ID does not exist" Oct 30 00:31:05.318830 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:05.318803 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v"] Oct 30 00:31:05.327676 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:05.327652 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-3fb3ad/ipp-route-3fb3ad-kserve-7bc74f8749-g5k2v"] Oct 30 00:31:05.378785 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:05.378758 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897"] Oct 30 00:31:06.920592 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:06.920558 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e" path="/var/lib/kubelet/pods/27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e/volumes" Oct 30 00:31:09.426111 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.426076 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8"] Oct 30 00:31:09.426760 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.426744 5001 cpu_manager.go:414] "containerMap: removing container" podUID="27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e" containerName="main" Oct 30 00:31:09.426828 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.426760 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e" containerName="main" Oct 30 00:31:09.426940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.426900 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="27fd701d-57b6-4ea6-a0ff-b8cb8acb5c5e" containerName="main" Oct 30 00:31:09.431886 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.431868 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:31:09.455095 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.455067 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8"] Oct 30 00:31:09.579313 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.579274 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:31:09.579486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.579330 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:31:09.579486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.579398 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:31:09.579486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.579464 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fddafb0-a134-4d5b-9956-0e9b925e799e-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:31:09.579599 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.579533 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fddafb0-a134-4d5b-9956-0e9b925e799e-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:31:09.579655 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.579631 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fddafb0-a134-4d5b-9956-0e9b925e799e-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:31:09.579765 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.579744 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:31:09.579804 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.579775 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5krcv\" (UniqueName: \"kubernetes.io/projected/9fddafb0-a134-4d5b-9956-0e9b925e799e-kube-api-access-5krcv\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:31:09.579849 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.579836 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:31:09.681179 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.681104 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fddafb0-a134-4d5b-9956-0e9b925e799e-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:31:09.681336 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.681180 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:31:09.681336 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.681200 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5krcv\" (UniqueName: \"kubernetes.io/projected/9fddafb0-a134-4d5b-9956-0e9b925e799e-kube-api-access-5krcv\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:31:09.681336 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.681233 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:31:09.681336 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.681300 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:31:09.681336 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.681330 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:31:09.681624 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.681444 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:31:09.681624 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.681531 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fddafb0-a134-4d5b-9956-0e9b925e799e-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:31:09.681624 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.681605 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-credential-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:31:09.681854 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.681700 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-workload-certs\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:31:09.681854 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.681727 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fddafb0-a134-4d5b-9956-0e9b925e799e-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:31:09.681854 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.681752 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-workload-socket\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:31:09.681854 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.681807 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-istio-data\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:31:09.682362 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.682337 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fddafb0-a134-4d5b-9956-0e9b925e799e-istiod-ca-cert\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:31:09.683619 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.683595 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-istio-envoy\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:31:09.684015 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.683999 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fddafb0-a134-4d5b-9956-0e9b925e799e-istio-podinfo\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:31:09.701859 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.701836 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5krcv\" (UniqueName: \"kubernetes.io/projected/9fddafb0-a134-4d5b-9956-0e9b925e799e-kube-api-access-5krcv\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:31:09.702005 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.701894 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fddafb0-a134-4d5b-9956-0e9b925e799e-istio-token\") pod \"e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:31:09.749271 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:09.749231 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:31:10.075025 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:31:10.074998 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fddafb0_a134_4d5b_9956_0e9b925e799e.slice/crio-46e3154296fe1cca93e3b959f11a931319653db44d24d70ae2458ce8fdb6661a WatchSource:0}: Error finding container 46e3154296fe1cca93e3b959f11a931319653db44d24d70ae2458ce8fdb6661a: Status 404 returned error can't find the container with id 46e3154296fe1cca93e3b959f11a931319653db44d24d70ae2458ce8fdb6661a Oct 30 00:31:10.075468 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:10.075446 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8"] Oct 30 00:31:10.079064 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:10.079028 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:31:10.079165 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:10.079097 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:31:10.079165 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:10.079130 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:31:10.327837 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:10.327749 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" event={"ID":"9fddafb0-a134-4d5b-9956-0e9b925e799e","Type":"ContainerStarted","Data":"3a36f9090689231879333f6fcbfb765b00b58f238cb11efaae899dddbca2ded7"} Oct 30 00:31:10.327837 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:10.327808 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" event={"ID":"9fddafb0-a134-4d5b-9956-0e9b925e799e","Type":"ContainerStarted","Data":"46e3154296fe1cca93e3b959f11a931319653db44d24d70ae2458ce8fdb6661a"} Oct 30 00:31:10.373321 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:10.373261 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" podStartSLOduration=1.373245807 podStartE2EDuration="1.373245807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:10.367202122 +0000 UTC m=+1585.572703303" watchObservedRunningTime="2026-10-30 00:31:10.373245807 +0000 UTC m=+1585.578746988" Oct 30 00:31:10.761063 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:10.758040 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:31:11.751255 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:11.751216 5001 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 00:31:11.751430 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:11.751280 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" podUID="9fddafb0-a134-4d5b-9956-0e9b925e799e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 00:31:12.749641 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:12.749608 5001 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 00:31:12.750063 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:12.749668 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" podUID="9fddafb0-a134-4d5b-9956-0e9b925e799e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 00:31:13.758806 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:13.754313 5001 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 00:31:13.758806 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:13.754372 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" podUID="9fddafb0-a134-4d5b-9956-0e9b925e799e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 00:31:14.750436 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:14.750400 5001 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 00:31:14.750631 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:14.750470 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" podUID="9fddafb0-a134-4d5b-9956-0e9b925e799e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 00:31:15.750538 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:15.750496 5001 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 00:31:15.751088 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:15.750569 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" podUID="9fddafb0-a134-4d5b-9956-0e9b925e799e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 00:31:16.750191 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:16.750161 5001 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 00:31:16.750341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:16.750223 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" podUID="9fddafb0-a134-4d5b-9956-0e9b925e799e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 00:31:17.325932 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:31:17.325121 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:31:18.164450 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:18.164416 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:31:18.171294 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:18.171266 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:31:18.180270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:18.180238 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:31:20.413939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:20.407236 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" podUID="ec207720-74f5-4146-93cc-c633e090a5c8" containerName="main" containerID="cri-o://490bd7fe56bdcdc42c260b44023919d340e9021fece84ceda155d1566d2885b5" gracePeriod=45 Oct 30 00:31:20.519141 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:20.519090 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" podUID="ec207720-74f5-4146-93cc-c633e090a5c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.201:8000/ready\": dial tcp 10.132.0.201:8000: connect: connection refused" Oct 30 00:31:21.272938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:21.261264 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb"] Oct 30 00:31:21.309071 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:21.307268 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" podUID="e29579fa-0beb-4dea-8cde-bf5cd16afb5d" containerName="istio-proxy" containerID="cri-o://2b9de37a96bc5d5e1d3867bfe8ed70d9be1d9ba00468ace2a905c399c76128bd" gracePeriod=30 Oct 30 00:31:21.336938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:21.332883 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc"] Oct 30 00:31:21.337407 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:21.337370 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" podUID="73a204ce-06bf-46b8-a99e-a26007c69e8b" containerName="istio-proxy" containerID="cri-o://67241d3224c936ed20a6649fe2c91a692ea423853d15354fa25174a9db40d090" gracePeriod=30 Oct 30 00:31:21.635370 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:21.603661 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ae68a59-h7r2h"] Oct 30 00:31:21.892015 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:21.887372 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ae68a59-h7r2h" Oct 30 00:31:21.921209 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:21.918446 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:31:21.921209 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:21.918731 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:31:21.921209 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:21.918809 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:31:21.921209 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:21.919618 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:31:22.092954 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.088544 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd5kw\" (UniqueName: \"kubernetes.io/projected/4a43cdd7-b9e2-4c02-b4c8-bf785b583213-kube-api-access-jd5kw\") pod \"maas-api-revoke-sub-9ae68a59-h7r2h\" (UID: \"4a43cdd7-b9e2-4c02-b4c8-bf785b583213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ae68a59-h7r2h" Oct 30 00:31:22.092954 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.088844 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a43cdd7-b9e2-4c02-b4c8-bf785b583213-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ae68a59-h7r2h\" (UID: \"4a43cdd7-b9e2-4c02-b4c8-bf785b583213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ae68a59-h7r2h" Oct 30 00:31:22.092954 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.089001 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a43cdd7-b9e2-4c02-b4c8-bf785b583213-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ae68a59-h7r2h\" (UID: \"4a43cdd7-b9e2-4c02-b4c8-bf785b583213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ae68a59-h7r2h" Oct 30 00:31:22.168074 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.166833 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" event={"ID":"ec207720-74f5-4146-93cc-c633e090a5c8","Type":"ContainerDied","Data":"490bd7fe56bdcdc42c260b44023919d340e9021fece84ceda155d1566d2885b5"} Oct 30 00:31:22.175037 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.174982 5001 generic.go:356] "Generic (PLEG): container finished" podID="ec207720-74f5-4146-93cc-c633e090a5c8" containerID="490bd7fe56bdcdc42c260b44023919d340e9021fece84ceda155d1566d2885b5" exitCode=0 Oct 30 00:31:22.192520 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.192453 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a43cdd7-b9e2-4c02-b4c8-bf785b583213-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ae68a59-h7r2h\" (UID: \"4a43cdd7-b9e2-4c02-b4c8-bf785b583213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ae68a59-h7r2h" Oct 30 00:31:22.195224 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.195191 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd5kw\" (UniqueName: \"kubernetes.io/projected/4a43cdd7-b9e2-4c02-b4c8-bf785b583213-kube-api-access-jd5kw\") pod \"maas-api-revoke-sub-9ae68a59-h7r2h\" (UID: \"4a43cdd7-b9e2-4c02-b4c8-bf785b583213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ae68a59-h7r2h" Oct 30 00:31:22.196577 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.196553 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a43cdd7-b9e2-4c02-b4c8-bf785b583213-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ae68a59-h7r2h\" (UID: \"4a43cdd7-b9e2-4c02-b4c8-bf785b583213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ae68a59-h7r2h" Oct 30 00:31:22.203963 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.203732 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a43cdd7-b9e2-4c02-b4c8-bf785b583213-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ae68a59-h7r2h\" (UID: \"4a43cdd7-b9e2-4c02-b4c8-bf785b583213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ae68a59-h7r2h" Oct 30 00:31:22.233981 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.231096 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ae68a59-h7r2h"] Oct 30 00:31:22.272212 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.272147 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd5kw\" (UniqueName: \"kubernetes.io/projected/4a43cdd7-b9e2-4c02-b4c8-bf785b583213-kube-api-access-jd5kw\") pod \"maas-api-revoke-sub-9ae68a59-h7r2h\" (UID: \"4a43cdd7-b9e2-4c02-b4c8-bf785b583213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ae68a59-h7r2h" Oct 30 00:31:22.272545 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.272512 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a43cdd7-b9e2-4c02-b4c8-bf785b583213-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ae68a59-h7r2h\" (UID: \"4a43cdd7-b9e2-4c02-b4c8-bf785b583213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ae68a59-h7r2h" Oct 30 00:31:22.275181 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.275110 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ae68a59-h7r2h" Oct 30 00:31:22.315652 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.315478 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw"] Oct 30 00:31:22.315824 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.315710 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" podUID="65d1bac8-837e-446b-9340-5fd94ba8be01" containerName="istio-proxy" containerID="cri-o://e8ec9fa3fefbbefeead300db070e78d1be471f973205314474bb1e50d53b0108" gracePeriod=30 Oct 30 00:31:22.346926 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.346887 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" Oct 30 00:31:22.413528 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.411478 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec207720-74f5-4146-93cc-c633e090a5c8-home\" (UniqueName: \"kubernetes.io/empty-dir/ec207720-74f5-4146-93cc-c633e090a5c8-home\") pod \"ec207720-74f5-4146-93cc-c633e090a5c8\" (UID: \"ec207720-74f5-4146-93cc-c633e090a5c8\") " Oct 30 00:31:22.413528 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.411582 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec207720-74f5-4146-93cc-c633e090a5c8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec207720-74f5-4146-93cc-c633e090a5c8-model-cache\") pod \"ec207720-74f5-4146-93cc-c633e090a5c8\" (UID: \"ec207720-74f5-4146-93cc-c633e090a5c8\") " Oct 30 00:31:22.413528 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.411697 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec207720-74f5-4146-93cc-c633e090a5c8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec207720-74f5-4146-93cc-c633e090a5c8-dshm\") pod \"ec207720-74f5-4146-93cc-c633e090a5c8\" (UID: \"ec207720-74f5-4146-93cc-c633e090a5c8\") " Oct 30 00:31:22.413528 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.411733 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec207720-74f5-4146-93cc-c633e090a5c8-kube-api-access-dcnd6\" (UniqueName: \"kubernetes.io/projected/ec207720-74f5-4146-93cc-c633e090a5c8-kube-api-access-dcnd6\") pod \"ec207720-74f5-4146-93cc-c633e090a5c8\" (UID: \"ec207720-74f5-4146-93cc-c633e090a5c8\") " Oct 30 00:31:22.413528 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.411805 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec207720-74f5-4146-93cc-c633e090a5c8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec207720-74f5-4146-93cc-c633e090a5c8-tmp-dir\") pod \"ec207720-74f5-4146-93cc-c633e090a5c8\" (UID: \"ec207720-74f5-4146-93cc-c633e090a5c8\") " Oct 30 00:31:22.413528 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.411901 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec207720-74f5-4146-93cc-c633e090a5c8-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec207720-74f5-4146-93cc-c633e090a5c8-tls-certs\") pod \"ec207720-74f5-4146-93cc-c633e090a5c8\" (UID: \"ec207720-74f5-4146-93cc-c633e090a5c8\") " Oct 30 00:31:22.413528 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.412685 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec207720-74f5-4146-93cc-c633e090a5c8-tmp-dir" pod "ec207720-74f5-4146-93cc-c633e090a5c8" (UID: "ec207720-74f5-4146-93cc-c633e090a5c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:22.413528 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.412702 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec207720-74f5-4146-93cc-c633e090a5c8-home" pod "ec207720-74f5-4146-93cc-c633e090a5c8" (UID: "ec207720-74f5-4146-93cc-c633e090a5c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:22.413528 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.412884 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec207720-74f5-4146-93cc-c633e090a5c8-model-cache" pod "ec207720-74f5-4146-93cc-c633e090a5c8" (UID: "ec207720-74f5-4146-93cc-c633e090a5c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:22.413528 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.413377 5001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec207720-74f5-4146-93cc-c633e090a5c8-home\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:22.413528 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.413399 5001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec207720-74f5-4146-93cc-c633e090a5c8-model-cache\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:22.413528 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.413415 5001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec207720-74f5-4146-93cc-c633e090a5c8-tmp-dir\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:22.415885 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.415856 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec207720-74f5-4146-93cc-c633e090a5c8-kube-api-access-dcnd6" pod "ec207720-74f5-4146-93cc-c633e090a5c8" (UID: "ec207720-74f5-4146-93cc-c633e090a5c8"). InnerVolumeSpecName "kube-api-access-dcnd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:31:22.416012 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.415994 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec207720-74f5-4146-93cc-c633e090a5c8-dshm" pod "ec207720-74f5-4146-93cc-c633e090a5c8" (UID: "ec207720-74f5-4146-93cc-c633e090a5c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:22.416207 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.416182 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec207720-74f5-4146-93cc-c633e090a5c8-tls-certs" pod "ec207720-74f5-4146-93cc-c633e090a5c8" (UID: "ec207720-74f5-4146-93cc-c633e090a5c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:31:22.519195 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.519112 5001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec207720-74f5-4146-93cc-c633e090a5c8-tls-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:22.519195 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.519152 5001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec207720-74f5-4146-93cc-c633e090a5c8-dshm\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:22.519195 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.519167 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcnd6\" (UniqueName: \"kubernetes.io/projected/ec207720-74f5-4146-93cc-c633e090a5c8-kube-api-access-dcnd6\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:22.536562 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.536528 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" podUID="ec207720-74f5-4146-93cc-c633e090a5c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.201:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:31:22.536721 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.536610 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" Oct 30 00:31:22.640725 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:22.640696 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ae68a59-h7r2h"] Oct 30 00:31:22.641433 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:31:22.641402 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a43cdd7_b9e2_4c02_b4c8_bf785b583213.slice/crio-b8d9fbee0919da5ba0e15f72bf934780ec0ce919219c570ae649c9ef21375628 WatchSource:0}: Error finding container b8d9fbee0919da5ba0e15f72bf934780ec0ce919219c570ae649c9ef21375628: Status 404 returned error can't find the container with id b8d9fbee0919da5ba0e15f72bf934780ec0ce919219c570ae649c9ef21375628 Oct 30 00:31:23.185619 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:23.185530 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ae68a59-h7r2h" event={"ID":"4a43cdd7-b9e2-4c02-b4c8-bf785b583213","Type":"ContainerStarted","Data":"4fc8ab6c8a3e9e175412fc0d2b99aead7879429f71ba3a90773aba9d3d7abe58"} Oct 30 00:31:23.185619 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:23.185587 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ae68a59-h7r2h" event={"ID":"4a43cdd7-b9e2-4c02-b4c8-bf785b583213","Type":"ContainerStarted","Data":"b8d9fbee0919da5ba0e15f72bf934780ec0ce919219c570ae649c9ef21375628"} Oct 30 00:31:23.186992 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:23.186970 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" Oct 30 00:31:23.187108 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:23.187005 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897" event={"ID":"ec207720-74f5-4146-93cc-c633e090a5c8","Type":"ContainerDied","Data":"9f1436332a2930c3b0bbacf05acc59e3980be01e156d7ba550054091823035c0"} Oct 30 00:31:23.187108 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:23.187055 5001 scope.go:122] "RemoveContainer" containerID="490bd7fe56bdcdc42c260b44023919d340e9021fece84ceda155d1566d2885b5" Oct 30 00:31:23.224165 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:23.224113 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ae68a59-h7r2h" podStartSLOduration=2.224096044 podStartE2EDuration="2.224096044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:23.219694496 +0000 UTC m=+1598.425195677" watchObservedRunningTime="2026-10-30 00:31:23.224096044 +0000 UTC m=+1598.429597228" Oct 30 00:31:23.254200 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:23.254173 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897"] Oct 30 00:31:23.258844 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:23.258821 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-93016b/praxis-route-93016b-kserve-f95f49c9-dn897"] Oct 30 00:31:24.199511 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:24.199478 5001 generic.go:356] "Generic (PLEG): container finished" podID="4a43cdd7-b9e2-4c02-b4c8-bf785b583213" containerID="4fc8ab6c8a3e9e175412fc0d2b99aead7879429f71ba3a90773aba9d3d7abe58" exitCode=0 Oct 30 00:31:24.200016 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:24.199545 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ae68a59-h7r2h" event={"ID":"4a43cdd7-b9e2-4c02-b4c8-bf785b583213","Type":"ContainerDied","Data":"4fc8ab6c8a3e9e175412fc0d2b99aead7879429f71ba3a90773aba9d3d7abe58"} Oct 30 00:31:24.931539 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:24.929566 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec207720-74f5-4146-93cc-c633e090a5c8" path="/var/lib/kubelet/pods/ec207720-74f5-4146-93cc-c633e090a5c8/volumes" Oct 30 00:31:25.377355 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:25.377326 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ae68a59-h7r2h" Oct 30 00:31:25.495778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:25.495701 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a43cdd7-b9e2-4c02-b4c8-bf785b583213-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a43cdd7-b9e2-4c02-b4c8-bf785b583213-maas-api-xks-ca\") pod \"4a43cdd7-b9e2-4c02-b4c8-bf785b583213\" (UID: \"4a43cdd7-b9e2-4c02-b4c8-bf785b583213\") " Oct 30 00:31:25.495951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:25.495878 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a43cdd7-b9e2-4c02-b4c8-bf785b583213-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a43cdd7-b9e2-4c02-b4c8-bf785b583213-maas-api-openshift-ca\") pod \"4a43cdd7-b9e2-4c02-b4c8-bf785b583213\" (UID: \"4a43cdd7-b9e2-4c02-b4c8-bf785b583213\") " Oct 30 00:31:25.496165 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:25.496145 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a43cdd7-b9e2-4c02-b4c8-bf785b583213-kube-api-access-jd5kw\" (UniqueName: \"kubernetes.io/projected/4a43cdd7-b9e2-4c02-b4c8-bf785b583213-kube-api-access-jd5kw\") pod \"4a43cdd7-b9e2-4c02-b4c8-bf785b583213\" (UID: \"4a43cdd7-b9e2-4c02-b4c8-bf785b583213\") " Oct 30 00:31:25.496256 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:25.496236 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a43cdd7-b9e2-4c02-b4c8-bf785b583213-maas-api-openshift-ca" pod "4a43cdd7-b9e2-4c02-b4c8-bf785b583213" (UID: "4a43cdd7-b9e2-4c02-b4c8-bf785b583213"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:31:25.496719 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:25.496696 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a43cdd7-b9e2-4c02-b4c8-bf785b583213-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:25.497747 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:25.497722 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a43cdd7-b9e2-4c02-b4c8-bf785b583213-maas-api-xks-ca" pod "4a43cdd7-b9e2-4c02-b4c8-bf785b583213" (UID: "4a43cdd7-b9e2-4c02-b4c8-bf785b583213"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:31:25.498108 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:25.498085 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a43cdd7-b9e2-4c02-b4c8-bf785b583213-kube-api-access-jd5kw" pod "4a43cdd7-b9e2-4c02-b4c8-bf785b583213" (UID: "4a43cdd7-b9e2-4c02-b4c8-bf785b583213"). InnerVolumeSpecName "kube-api-access-jd5kw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:31:25.598458 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:25.598428 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd5kw\" (UniqueName: \"kubernetes.io/projected/4a43cdd7-b9e2-4c02-b4c8-bf785b583213-kube-api-access-jd5kw\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:25.598458 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:25.598457 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a43cdd7-b9e2-4c02-b4c8-bf785b583213-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:26.398935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.398760 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ae68a59-h7r2h" event={"ID":"4a43cdd7-b9e2-4c02-b4c8-bf785b583213","Type":"ContainerDied","Data":"b8d9fbee0919da5ba0e15f72bf934780ec0ce919219c570ae649c9ef21375628"} Oct 30 00:31:26.398935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.398828 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8d9fbee0919da5ba0e15f72bf934780ec0ce919219c570ae649c9ef21375628" Oct 30 00:31:26.398935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.398896 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ae68a59-h7r2h" Oct 30 00:31:26.740649 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.740606 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:31:26.740835 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.740682 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" podUID="e29579fa-0beb-4dea-8cde-bf5cd16afb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 00:31:26.844616 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.844580 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:31:26.887288 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.887263 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:31:26.896177 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.896144 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73a204ce-06bf-46b8-a99e-a26007c69e8b-kube-api-access-l9vp9\" (UniqueName: \"kubernetes.io/projected/73a204ce-06bf-46b8-a99e-a26007c69e8b-kube-api-access-l9vp9\") pod \"73a204ce-06bf-46b8-a99e-a26007c69e8b\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " Oct 30 00:31:26.896349 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.896244 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-workload-socket\") pod \"73a204ce-06bf-46b8-a99e-a26007c69e8b\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " Oct 30 00:31:26.896349 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.896346 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73a204ce-06bf-46b8-a99e-a26007c69e8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73a204ce-06bf-46b8-a99e-a26007c69e8b-istio-podinfo\") pod \"73a204ce-06bf-46b8-a99e-a26007c69e8b\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " Oct 30 00:31:26.896467 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.896409 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-credential-socket\") pod \"73a204ce-06bf-46b8-a99e-a26007c69e8b\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " Oct 30 00:31:26.896467 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.896448 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73a204ce-06bf-46b8-a99e-a26007c69e8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73a204ce-06bf-46b8-a99e-a26007c69e8b-istiod-ca-cert\") pod \"73a204ce-06bf-46b8-a99e-a26007c69e8b\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " Oct 30 00:31:26.896586 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.896477 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-istio-data\") pod \"73a204ce-06bf-46b8-a99e-a26007c69e8b\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " Oct 30 00:31:26.896586 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.896555 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-istio-envoy\") pod \"73a204ce-06bf-46b8-a99e-a26007c69e8b\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " Oct 30 00:31:26.896695 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.896593 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73a204ce-06bf-46b8-a99e-a26007c69e8b-istio-token\" (UniqueName: \"kubernetes.io/projected/73a204ce-06bf-46b8-a99e-a26007c69e8b-istio-token\") pod \"73a204ce-06bf-46b8-a99e-a26007c69e8b\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " Oct 30 00:31:26.896749 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.896733 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-workload-certs\") pod \"73a204ce-06bf-46b8-a99e-a26007c69e8b\" (UID: \"73a204ce-06bf-46b8-a99e-a26007c69e8b\") " Oct 30 00:31:26.897281 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.897253 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-workload-certs" pod "73a204ce-06bf-46b8-a99e-a26007c69e8b" (UID: "73a204ce-06bf-46b8-a99e-a26007c69e8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:26.898081 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.898055 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:26.898622 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.898290 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73a204ce-06bf-46b8-a99e-a26007c69e8b-istiod-ca-cert" pod "73a204ce-06bf-46b8-a99e-a26007c69e8b" (UID: "73a204ce-06bf-46b8-a99e-a26007c69e8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:31:26.898622 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.898532 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-workload-socket" pod "73a204ce-06bf-46b8-a99e-a26007c69e8b" (UID: "73a204ce-06bf-46b8-a99e-a26007c69e8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:26.900407 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.900376 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73a204ce-06bf-46b8-a99e-a26007c69e8b-kube-api-access-l9vp9" pod "73a204ce-06bf-46b8-a99e-a26007c69e8b" (UID: "73a204ce-06bf-46b8-a99e-a26007c69e8b"). InnerVolumeSpecName "kube-api-access-l9vp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:31:26.900631 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.900606 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-istio-envoy" pod "73a204ce-06bf-46b8-a99e-a26007c69e8b" (UID: "73a204ce-06bf-46b8-a99e-a26007c69e8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:26.900833 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.900812 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-istio-data" pod "73a204ce-06bf-46b8-a99e-a26007c69e8b" (UID: "73a204ce-06bf-46b8-a99e-a26007c69e8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:26.900965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.900834 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73a204ce-06bf-46b8-a99e-a26007c69e8b-istio-podinfo" pod "73a204ce-06bf-46b8-a99e-a26007c69e8b" (UID: "73a204ce-06bf-46b8-a99e-a26007c69e8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:31:26.901136 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.901109 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-credential-socket" pod "73a204ce-06bf-46b8-a99e-a26007c69e8b" (UID: "73a204ce-06bf-46b8-a99e-a26007c69e8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:26.902834 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.902802 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73a204ce-06bf-46b8-a99e-a26007c69e8b-istio-token" pod "73a204ce-06bf-46b8-a99e-a26007c69e8b" (UID: "73a204ce-06bf-46b8-a99e-a26007c69e8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:31:26.998772 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.998683 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-credential-socket\") pod \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " Oct 30 00:31:26.998978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.998772 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-workload-certs\") pod \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " Oct 30 00:31:26.998978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.998824 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istiod-ca-cert\") pod \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " Oct 30 00:31:26.998978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.998904 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istio-podinfo\") pod \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " Oct 30 00:31:26.999179 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.998976 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-credential-socket" pod "e29579fa-0beb-4dea-8cde-bf5cd16afb5d" (UID: "e29579fa-0beb-4dea-8cde-bf5cd16afb5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:26.999179 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.999068 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istio-envoy\") pod \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " Oct 30 00:31:26.999291 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.999223 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istio-token\" (UniqueName: \"kubernetes.io/projected/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istio-token\") pod \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " Oct 30 00:31:26.999291 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.999254 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-workload-socket\") pod \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " Oct 30 00:31:26.999402 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.999301 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istio-data\") pod \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " Oct 30 00:31:26.999464 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.999400 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-kube-api-access-rgvdx\" (UniqueName: \"kubernetes.io/projected/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-kube-api-access-rgvdx\") pod \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\" (UID: \"e29579fa-0beb-4dea-8cde-bf5cd16afb5d\") " Oct 30 00:31:26.999521 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.999501 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-workload-socket" pod "e29579fa-0beb-4dea-8cde-bf5cd16afb5d" (UID: "e29579fa-0beb-4dea-8cde-bf5cd16afb5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:26.999716 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.999675 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istiod-ca-cert" pod "e29579fa-0beb-4dea-8cde-bf5cd16afb5d" (UID: "e29579fa-0beb-4dea-8cde-bf5cd16afb5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:31:26.999716 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:26.999708 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istio-data" pod "e29579fa-0beb-4dea-8cde-bf5cd16afb5d" (UID: "e29579fa-0beb-4dea-8cde-bf5cd16afb5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:27.000700 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.000007 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-workload-certs" pod "e29579fa-0beb-4dea-8cde-bf5cd16afb5d" (UID: "e29579fa-0beb-4dea-8cde-bf5cd16afb5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:27.000700 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.000263 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:27.000700 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.000290 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:27.000700 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.000304 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73a204ce-06bf-46b8-a99e-a26007c69e8b-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:27.000700 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.000319 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:27.000700 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.000332 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:27.000700 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.000346 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73a204ce-06bf-46b8-a99e-a26007c69e8b-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:27.000700 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.000359 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:27.000700 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.000371 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:27.000700 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.000384 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:27.000700 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.000395 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:27.000700 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.000408 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73a204ce-06bf-46b8-a99e-a26007c69e8b-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:27.000700 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.000421 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9vp9\" (UniqueName: \"kubernetes.io/projected/73a204ce-06bf-46b8-a99e-a26007c69e8b-kube-api-access-l9vp9\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:27.000700 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.000435 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a204ce-06bf-46b8-a99e-a26007c69e8b-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:27.002037 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.002011 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-kube-api-access-rgvdx" pod "e29579fa-0beb-4dea-8cde-bf5cd16afb5d" (UID: "e29579fa-0beb-4dea-8cde-bf5cd16afb5d"). InnerVolumeSpecName "kube-api-access-rgvdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:31:27.002263 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.002240 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istio-envoy" pod "e29579fa-0beb-4dea-8cde-bf5cd16afb5d" (UID: "e29579fa-0beb-4dea-8cde-bf5cd16afb5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:27.002380 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.002267 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istio-token" pod "e29579fa-0beb-4dea-8cde-bf5cd16afb5d" (UID: "e29579fa-0beb-4dea-8cde-bf5cd16afb5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:31:27.003578 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.003557 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istio-podinfo" pod "e29579fa-0beb-4dea-8cde-bf5cd16afb5d" (UID: "e29579fa-0beb-4dea-8cde-bf5cd16afb5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:31:27.101998 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.101966 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:27.101998 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.101996 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:27.102200 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.102009 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgvdx\" (UniqueName: \"kubernetes.io/projected/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-kube-api-access-rgvdx\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:27.102200 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.102023 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e29579fa-0beb-4dea-8cde-bf5cd16afb5d-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:27.409343 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.409313 5001 generic.go:356] "Generic (PLEG): container finished" podID="73a204ce-06bf-46b8-a99e-a26007c69e8b" containerID="67241d3224c936ed20a6649fe2c91a692ea423853d15354fa25174a9db40d090" exitCode=0 Oct 30 00:31:27.409832 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.409356 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" Oct 30 00:31:27.409832 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.409398 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" event={"ID":"73a204ce-06bf-46b8-a99e-a26007c69e8b","Type":"ContainerDied","Data":"67241d3224c936ed20a6649fe2c91a692ea423853d15354fa25174a9db40d090"} Oct 30 00:31:27.409832 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.409454 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc" event={"ID":"73a204ce-06bf-46b8-a99e-a26007c69e8b","Type":"ContainerDied","Data":"bd99b24c239006c0c96169fc74a544d2ca0ef2b4f2fc0ff320c6f4200281065f"} Oct 30 00:31:27.409832 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.409481 5001 scope.go:122] "RemoveContainer" containerID="67241d3224c936ed20a6649fe2c91a692ea423853d15354fa25174a9db40d090" Oct 30 00:31:27.411295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.411273 5001 generic.go:356] "Generic (PLEG): container finished" podID="e29579fa-0beb-4dea-8cde-bf5cd16afb5d" containerID="2b9de37a96bc5d5e1d3867bfe8ed70d9be1d9ba00468ace2a905c399c76128bd" exitCode=0 Oct 30 00:31:27.411435 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.411303 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" event={"ID":"e29579fa-0beb-4dea-8cde-bf5cd16afb5d","Type":"ContainerDied","Data":"2b9de37a96bc5d5e1d3867bfe8ed70d9be1d9ba00468ace2a905c399c76128bd"} Oct 30 00:31:27.411435 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.411315 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" Oct 30 00:31:27.411435 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.411333 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb" event={"ID":"e29579fa-0beb-4dea-8cde-bf5cd16afb5d","Type":"ContainerDied","Data":"7e96a29e9439e0361b1688914adef84e54b33a520e973fe0884f37bda5858a69"} Oct 30 00:31:27.445087 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.445064 5001 scope.go:122] "RemoveContainer" containerID="67241d3224c936ed20a6649fe2c91a692ea423853d15354fa25174a9db40d090" Oct 30 00:31:27.445774 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:31:27.445737 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67241d3224c936ed20a6649fe2c91a692ea423853d15354fa25174a9db40d090\": container with ID starting with 67241d3224c936ed20a6649fe2c91a692ea423853d15354fa25174a9db40d090 not found: ID does not exist" containerID="67241d3224c936ed20a6649fe2c91a692ea423853d15354fa25174a9db40d090" Oct 30 00:31:27.445895 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.445782 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67241d3224c936ed20a6649fe2c91a692ea423853d15354fa25174a9db40d090"} err="failed to get container status \"67241d3224c936ed20a6649fe2c91a692ea423853d15354fa25174a9db40d090\": rpc error: code = NotFound desc = could not find container \"67241d3224c936ed20a6649fe2c91a692ea423853d15354fa25174a9db40d090\": container with ID starting with 67241d3224c936ed20a6649fe2c91a692ea423853d15354fa25174a9db40d090 not found: ID does not exist" Oct 30 00:31:27.445895 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.445805 5001 scope.go:122] "RemoveContainer" containerID="2b9de37a96bc5d5e1d3867bfe8ed70d9be1d9ba00468ace2a905c399c76128bd" Oct 30 00:31:27.457631 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.457613 5001 scope.go:122] "RemoveContainer" containerID="2b9de37a96bc5d5e1d3867bfe8ed70d9be1d9ba00468ace2a905c399c76128bd" Oct 30 00:31:27.457716 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.457640 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb"] Oct 30 00:31:27.457979 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:31:27.457898 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b9de37a96bc5d5e1d3867bfe8ed70d9be1d9ba00468ace2a905c399c76128bd\": container with ID starting with 2b9de37a96bc5d5e1d3867bfe8ed70d9be1d9ba00468ace2a905c399c76128bd not found: ID does not exist" containerID="2b9de37a96bc5d5e1d3867bfe8ed70d9be1d9ba00468ace2a905c399c76128bd" Oct 30 00:31:27.458045 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.457985 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b9de37a96bc5d5e1d3867bfe8ed70d9be1d9ba00468ace2a905c399c76128bd"} err="failed to get container status \"2b9de37a96bc5d5e1d3867bfe8ed70d9be1d9ba00468ace2a905c399c76128bd\": rpc error: code = NotFound desc = could not find container \"2b9de37a96bc5d5e1d3867bfe8ed70d9be1d9ba00468ace2a905c399c76128bd\": container with ID starting with 2b9de37a96bc5d5e1d3867bfe8ed70d9be1d9ba00468ace2a905c399c76128bd not found: ID does not exist" Oct 30 00:31:27.470544 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.470522 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-bnrfb"] Oct 30 00:31:27.493628 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.493588 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc"] Oct 30 00:31:27.499386 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.499358 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nwgrc"] Oct 30 00:31:27.642059 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.642032 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:31:27.711339 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.711305 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65d1bac8-837e-446b-9340-5fd94ba8be01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65d1bac8-837e-446b-9340-5fd94ba8be01-istiod-ca-cert\") pod \"65d1bac8-837e-446b-9340-5fd94ba8be01\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " Oct 30 00:31:27.711542 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.711409 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-istio-data\") pod \"65d1bac8-837e-446b-9340-5fd94ba8be01\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " Oct 30 00:31:27.711613 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.711559 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-workload-certs\") pod \"65d1bac8-837e-446b-9340-5fd94ba8be01\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " Oct 30 00:31:27.711672 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.711640 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65d1bac8-837e-446b-9340-5fd94ba8be01-kube-api-access-d8sbl\" (UniqueName: \"kubernetes.io/projected/65d1bac8-837e-446b-9340-5fd94ba8be01-kube-api-access-d8sbl\") pod \"65d1bac8-837e-446b-9340-5fd94ba8be01\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " Oct 30 00:31:27.711743 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.711713 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65d1bac8-837e-446b-9340-5fd94ba8be01-istiod-ca-cert" pod "65d1bac8-837e-446b-9340-5fd94ba8be01" (UID: "65d1bac8-837e-446b-9340-5fd94ba8be01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:31:27.711809 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.711735 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-credential-socket\") pod \"65d1bac8-837e-446b-9340-5fd94ba8be01\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " Oct 30 00:31:27.711866 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.711826 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65d1bac8-837e-446b-9340-5fd94ba8be01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65d1bac8-837e-446b-9340-5fd94ba8be01-istio-podinfo\") pod \"65d1bac8-837e-446b-9340-5fd94ba8be01\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " Oct 30 00:31:27.711944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.711869 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-workload-socket\") pod \"65d1bac8-837e-446b-9340-5fd94ba8be01\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " Oct 30 00:31:27.712041 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.712018 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-credential-socket" pod "65d1bac8-837e-446b-9340-5fd94ba8be01" (UID: "65d1bac8-837e-446b-9340-5fd94ba8be01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:27.712191 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.712101 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-istio-envoy\") pod \"65d1bac8-837e-446b-9340-5fd94ba8be01\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " Oct 30 00:31:27.712344 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.712237 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-workload-socket" pod "65d1bac8-837e-446b-9340-5fd94ba8be01" (UID: "65d1bac8-837e-446b-9340-5fd94ba8be01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:27.712344 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.712239 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-workload-certs" pod "65d1bac8-837e-446b-9340-5fd94ba8be01" (UID: "65d1bac8-837e-446b-9340-5fd94ba8be01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:27.712344 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.712276 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65d1bac8-837e-446b-9340-5fd94ba8be01-istio-token\" (UniqueName: \"kubernetes.io/projected/65d1bac8-837e-446b-9340-5fd94ba8be01-istio-token\") pod \"65d1bac8-837e-446b-9340-5fd94ba8be01\" (UID: \"65d1bac8-837e-446b-9340-5fd94ba8be01\") " Oct 30 00:31:27.712344 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.712274 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-istio-data" pod "65d1bac8-837e-446b-9340-5fd94ba8be01" (UID: "65d1bac8-837e-446b-9340-5fd94ba8be01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:27.713073 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.713047 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:27.713258 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.713078 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:27.713258 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.713093 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:27.713258 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.713107 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65d1bac8-837e-446b-9340-5fd94ba8be01-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:27.713258 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.713121 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:27.714581 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.714527 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65d1bac8-837e-446b-9340-5fd94ba8be01-kube-api-access-d8sbl" pod "65d1bac8-837e-446b-9340-5fd94ba8be01" (UID: "65d1bac8-837e-446b-9340-5fd94ba8be01"). InnerVolumeSpecName "kube-api-access-d8sbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:31:27.714833 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.714806 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65d1bac8-837e-446b-9340-5fd94ba8be01-istio-token" pod "65d1bac8-837e-446b-9340-5fd94ba8be01" (UID: "65d1bac8-837e-446b-9340-5fd94ba8be01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:31:27.723267 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.723237 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65d1bac8-837e-446b-9340-5fd94ba8be01-istio-podinfo" pod "65d1bac8-837e-446b-9340-5fd94ba8be01" (UID: "65d1bac8-837e-446b-9340-5fd94ba8be01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:31:27.723378 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.723327 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-istio-envoy" pod "65d1bac8-837e-446b-9340-5fd94ba8be01" (UID: "65d1bac8-837e-446b-9340-5fd94ba8be01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:27.827939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.820308 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8sbl\" (UniqueName: \"kubernetes.io/projected/65d1bac8-837e-446b-9340-5fd94ba8be01-kube-api-access-d8sbl\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:27.827939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.820348 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65d1bac8-837e-446b-9340-5fd94ba8be01-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:27.827939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.820365 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65d1bac8-837e-446b-9340-5fd94ba8be01-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:27.827939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:27.820379 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65d1bac8-837e-446b-9340-5fd94ba8be01-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:28.437841 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:28.437800 5001 generic.go:356] "Generic (PLEG): container finished" podID="65d1bac8-837e-446b-9340-5fd94ba8be01" containerID="e8ec9fa3fefbbefeead300db070e78d1be471f973205314474bb1e50d53b0108" exitCode=0 Oct 30 00:31:28.438301 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:28.437865 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" Oct 30 00:31:28.438301 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:28.437928 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" event={"ID":"65d1bac8-837e-446b-9340-5fd94ba8be01","Type":"ContainerDied","Data":"e8ec9fa3fefbbefeead300db070e78d1be471f973205314474bb1e50d53b0108"} Oct 30 00:31:28.438301 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:28.438006 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw" event={"ID":"65d1bac8-837e-446b-9340-5fd94ba8be01","Type":"ContainerDied","Data":"8ccc8b325cda184872391d91eb7e09d8da4ea6cee8692e5accd3872e2d5aa7b1"} Oct 30 00:31:28.438301 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:28.438036 5001 scope.go:122] "RemoveContainer" containerID="e8ec9fa3fefbbefeead300db070e78d1be471f973205314474bb1e50d53b0108" Oct 30 00:31:28.451056 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:28.451037 5001 scope.go:122] "RemoveContainer" containerID="e8ec9fa3fefbbefeead300db070e78d1be471f973205314474bb1e50d53b0108" Oct 30 00:31:28.451305 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:31:28.451286 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8ec9fa3fefbbefeead300db070e78d1be471f973205314474bb1e50d53b0108\": container with ID starting with e8ec9fa3fefbbefeead300db070e78d1be471f973205314474bb1e50d53b0108 not found: ID does not exist" containerID="e8ec9fa3fefbbefeead300db070e78d1be471f973205314474bb1e50d53b0108" Oct 30 00:31:28.451401 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:28.451313 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ec9fa3fefbbefeead300db070e78d1be471f973205314474bb1e50d53b0108"} err="failed to get container status \"e8ec9fa3fefbbefeead300db070e78d1be471f973205314474bb1e50d53b0108\": rpc error: code = NotFound desc = could not find container \"e8ec9fa3fefbbefeead300db070e78d1be471f973205314474bb1e50d53b0108\": container with ID starting with e8ec9fa3fefbbefeead300db070e78d1be471f973205314474bb1e50d53b0108 not found: ID does not exist" Oct 30 00:31:28.476180 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:31:28.476148 5001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d1bac8_837e_446b_9340_5fd94ba8be01.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d1bac8_837e_446b_9340_5fd94ba8be01.slice/crio-8ccc8b325cda184872391d91eb7e09d8da4ea6cee8692e5accd3872e2d5aa7b1\": RecentStats: unable to find data in memory cache]" Oct 30 00:31:28.495561 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:28.495528 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw"] Oct 30 00:31:28.504900 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:28.504874 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-22xjw"] Oct 30 00:31:28.920339 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:28.920309 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65d1bac8-837e-446b-9340-5fd94ba8be01" path="/var/lib/kubelet/pods/65d1bac8-837e-446b-9340-5fd94ba8be01/volumes" Oct 30 00:31:28.920873 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:28.920858 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73a204ce-06bf-46b8-a99e-a26007c69e8b" path="/var/lib/kubelet/pods/73a204ce-06bf-46b8-a99e-a26007c69e8b/volumes" Oct 30 00:31:28.921426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:28.921410 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e29579fa-0beb-4dea-8cde-bf5cd16afb5d" path="/var/lib/kubelet/pods/e29579fa-0beb-4dea-8cde-bf5cd16afb5d/volumes" Oct 30 00:31:31.295197 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:31.295158 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd"] Oct 30 00:31:31.295649 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:31.295357 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" podUID="a5ff3159-b9e7-4865-b35a-c2051d7e5c20" containerName="istio-proxy" containerID="cri-o://858956fd8fd665f36b2412e601621014cccbdbd473f21c4f2fdf5ce73b7a894c" gracePeriod=30 Oct 30 00:31:31.305318 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:31.305286 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm"] Oct 30 00:31:31.305582 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:31.305523 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" podUID="922f38ba-e017-4005-9841-aa0fd46b1d3c" containerName="istio-proxy" containerID="cri-o://b7fafa059445eb313e1005939bca2264a2ff067f01055e27dd2dc3fd1538a6a5" gracePeriod=30 Oct 30 00:31:33.884346 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:33.884296 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm"] Oct 30 00:31:33.884863 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:33.884510 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" podUID="44db3a17-452a-4667-bbc8-0f73b1711233" containerName="istio-proxy" containerID="cri-o://a67316a0d7da125cb9b9eaee0e3e791f08ebc0afda6326926f7a9bd19799a8c5" gracePeriod=30 Oct 30 00:31:34.409483 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:34.409445 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f649b0-4pgxm"] Oct 30 00:31:34.410152 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:34.410136 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4a43cdd7-b9e2-4c02-b4c8-bf785b583213" containerName="revoke-keys" Oct 30 00:31:34.410216 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:34.410154 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a43cdd7-b9e2-4c02-b4c8-bf785b583213" containerName="revoke-keys" Oct 30 00:31:34.410216 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:34.410169 5001 cpu_manager.go:414] "containerMap: removing container" podUID="ec207720-74f5-4146-93cc-c633e090a5c8" containerName="main" Oct 30 00:31:34.410216 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:34.410174 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec207720-74f5-4146-93cc-c633e090a5c8" containerName="main" Oct 30 00:31:34.410216 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:34.410189 5001 cpu_manager.go:414] "containerMap: removing container" podUID="65d1bac8-837e-446b-9340-5fd94ba8be01" containerName="istio-proxy" Oct 30 00:31:34.410216 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:34.410194 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65d1bac8-837e-446b-9340-5fd94ba8be01" containerName="istio-proxy" Oct 30 00:31:34.410216 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:34.410210 5001 cpu_manager.go:414] "containerMap: removing container" podUID="73a204ce-06bf-46b8-a99e-a26007c69e8b" containerName="istio-proxy" Oct 30 00:31:34.410216 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:34.410214 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73a204ce-06bf-46b8-a99e-a26007c69e8b" containerName="istio-proxy" Oct 30 00:31:34.410437 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:34.410231 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e29579fa-0beb-4dea-8cde-bf5cd16afb5d" containerName="istio-proxy" Oct 30 00:31:34.410437 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:34.410236 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e29579fa-0beb-4dea-8cde-bf5cd16afb5d" containerName="istio-proxy" Oct 30 00:31:34.410437 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:34.410331 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec207720-74f5-4146-93cc-c633e090a5c8" containerName="main" Oct 30 00:31:34.410437 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:34.410341 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="65d1bac8-837e-446b-9340-5fd94ba8be01" containerName="istio-proxy" Oct 30 00:31:34.410437 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:34.410348 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="73a204ce-06bf-46b8-a99e-a26007c69e8b" containerName="istio-proxy" Oct 30 00:31:34.410437 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:34.410356 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e29579fa-0beb-4dea-8cde-bf5cd16afb5d" containerName="istio-proxy" Oct 30 00:31:34.410437 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:34.410364 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a43cdd7-b9e2-4c02-b4c8-bf785b583213" containerName="revoke-keys" Oct 30 00:31:34.415431 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:34.415414 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f649b0-4pgxm" Oct 30 00:31:34.430480 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:34.430449 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:31:34.430634 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:34.430570 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:31:34.438246 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:34.438211 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f649b0-4pgxm"] Oct 30 00:31:34.455580 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:34.455549 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:31:34.455806 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:34.455593 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:31:34.554224 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:34.554192 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdv8s\" (UniqueName: \"kubernetes.io/projected/eda1038b-1cc5-4a86-934c-7c56e31be84b-kube-api-access-sdv8s\") pod \"maas-api-revoke-keys-e2e-ipp-b-f649b0-4pgxm\" (UID: \"eda1038b-1cc5-4a86-934c-7c56e31be84b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f649b0-4pgxm" Oct 30 00:31:34.554432 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:34.554410 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eda1038b-1cc5-4a86-934c-7c56e31be84b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f649b0-4pgxm\" (UID: \"eda1038b-1cc5-4a86-934c-7c56e31be84b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f649b0-4pgxm" Oct 30 00:31:34.554548 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:34.554530 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eda1038b-1cc5-4a86-934c-7c56e31be84b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f649b0-4pgxm\" (UID: \"eda1038b-1cc5-4a86-934c-7c56e31be84b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f649b0-4pgxm" Oct 30 00:31:34.655801 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:34.655770 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eda1038b-1cc5-4a86-934c-7c56e31be84b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f649b0-4pgxm\" (UID: \"eda1038b-1cc5-4a86-934c-7c56e31be84b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f649b0-4pgxm" Oct 30 00:31:34.655999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:34.655838 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eda1038b-1cc5-4a86-934c-7c56e31be84b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f649b0-4pgxm\" (UID: \"eda1038b-1cc5-4a86-934c-7c56e31be84b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f649b0-4pgxm" Oct 30 00:31:34.655999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:34.655977 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdv8s\" (UniqueName: \"kubernetes.io/projected/eda1038b-1cc5-4a86-934c-7c56e31be84b-kube-api-access-sdv8s\") pod \"maas-api-revoke-keys-e2e-ipp-b-f649b0-4pgxm\" (UID: \"eda1038b-1cc5-4a86-934c-7c56e31be84b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f649b0-4pgxm" Oct 30 00:31:34.656577 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:34.656553 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eda1038b-1cc5-4a86-934c-7c56e31be84b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f649b0-4pgxm\" (UID: \"eda1038b-1cc5-4a86-934c-7c56e31be84b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f649b0-4pgxm" Oct 30 00:31:34.658220 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:34.658190 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eda1038b-1cc5-4a86-934c-7c56e31be84b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f649b0-4pgxm\" (UID: \"eda1038b-1cc5-4a86-934c-7c56e31be84b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f649b0-4pgxm" Oct 30 00:31:34.678839 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:34.678777 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdv8s\" (UniqueName: \"kubernetes.io/projected/eda1038b-1cc5-4a86-934c-7c56e31be84b-kube-api-access-sdv8s\") pod \"maas-api-revoke-keys-e2e-ipp-b-f649b0-4pgxm\" (UID: \"eda1038b-1cc5-4a86-934c-7c56e31be84b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f649b0-4pgxm" Oct 30 00:31:34.730366 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:34.730338 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f649b0-4pgxm" Oct 30 00:31:35.039723 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:31:35.039694 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda1038b_1cc5_4a86_934c_7c56e31be84b.slice/crio-6d54995e0573da9839f0056bc5602ba87a20e3cbfe1a57deec9a7da78999c68f WatchSource:0}: Error finding container 6d54995e0573da9839f0056bc5602ba87a20e3cbfe1a57deec9a7da78999c68f: Status 404 returned error can't find the container with id 6d54995e0573da9839f0056bc5602ba87a20e3cbfe1a57deec9a7da78999c68f Oct 30 00:31:35.040159 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:35.040051 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f649b0-4pgxm"] Oct 30 00:31:35.519461 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:35.519428 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f649b0-4pgxm" event={"ID":"eda1038b-1cc5-4a86-934c-7c56e31be84b","Type":"ContainerStarted","Data":"9427d368df6f5ffd1d0a2f8be9f4336d79060fb44f7baeedb7259ffc70e4ee89"} Oct 30 00:31:35.519663 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:35.519491 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f649b0-4pgxm" event={"ID":"eda1038b-1cc5-4a86-934c-7c56e31be84b","Type":"ContainerStarted","Data":"6d54995e0573da9839f0056bc5602ba87a20e3cbfe1a57deec9a7da78999c68f"} Oct 30 00:31:35.556014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:35.555968 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f649b0-4pgxm" podStartSLOduration=1.555954966 podStartE2EDuration="1.555954966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:31:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:35.55397888 +0000 UTC m=+1610.759480060" watchObservedRunningTime="2026-10-30 00:31:35.555954966 +0000 UTC m=+1610.761456141" Oct 30 00:31:35.739872 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:35.739792 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c26df62a-8vccg"] Oct 30 00:31:35.744440 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:35.744416 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26df62a-8vccg" Oct 30 00:31:35.765398 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:35.765368 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c26df62a-8vccg"] Oct 30 00:31:35.879823 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:35.879790 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8wdr\" (UniqueName: \"kubernetes.io/projected/405f5b62-eb46-429f-9f68-9f444812c478-kube-api-access-g8wdr\") pod \"maas-api-revoke-sub-c26df62a-8vccg\" (UID: \"405f5b62-eb46-429f-9f68-9f444812c478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26df62a-8vccg" Oct 30 00:31:35.880036 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:35.879986 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405f5b62-eb46-429f-9f68-9f444812c478-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c26df62a-8vccg\" (UID: \"405f5b62-eb46-429f-9f68-9f444812c478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26df62a-8vccg" Oct 30 00:31:35.880036 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:35.880031 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405f5b62-eb46-429f-9f68-9f444812c478-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c26df62a-8vccg\" (UID: \"405f5b62-eb46-429f-9f68-9f444812c478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26df62a-8vccg" Oct 30 00:31:35.981551 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:35.981503 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8wdr\" (UniqueName: \"kubernetes.io/projected/405f5b62-eb46-429f-9f68-9f444812c478-kube-api-access-g8wdr\") pod \"maas-api-revoke-sub-c26df62a-8vccg\" (UID: \"405f5b62-eb46-429f-9f68-9f444812c478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26df62a-8vccg" Oct 30 00:31:35.981736 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:35.981666 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405f5b62-eb46-429f-9f68-9f444812c478-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c26df62a-8vccg\" (UID: \"405f5b62-eb46-429f-9f68-9f444812c478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26df62a-8vccg" Oct 30 00:31:35.981736 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:35.981722 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405f5b62-eb46-429f-9f68-9f444812c478-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c26df62a-8vccg\" (UID: \"405f5b62-eb46-429f-9f68-9f444812c478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26df62a-8vccg" Oct 30 00:31:35.982543 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:35.982516 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405f5b62-eb46-429f-9f68-9f444812c478-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c26df62a-8vccg\" (UID: \"405f5b62-eb46-429f-9f68-9f444812c478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26df62a-8vccg" Oct 30 00:31:35.984170 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:35.984143 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405f5b62-eb46-429f-9f68-9f444812c478-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c26df62a-8vccg\" (UID: \"405f5b62-eb46-429f-9f68-9f444812c478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26df62a-8vccg" Oct 30 00:31:36.002523 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:36.002469 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8wdr\" (UniqueName: \"kubernetes.io/projected/405f5b62-eb46-429f-9f68-9f444812c478-kube-api-access-g8wdr\") pod \"maas-api-revoke-sub-c26df62a-8vccg\" (UID: \"405f5b62-eb46-429f-9f68-9f444812c478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26df62a-8vccg" Oct 30 00:31:36.059136 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:36.059107 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26df62a-8vccg" Oct 30 00:31:36.371032 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:36.370174 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c26df62a-8vccg"] Oct 30 00:31:36.395775 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:31:36.395733 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405f5b62_eb46_429f_9f68_9f444812c478.slice/crio-ff6780f67ba639c0b28b4aa1847ed13dd1e88a26f3428086fae01488668589c1 WatchSource:0}: Error finding container ff6780f67ba639c0b28b4aa1847ed13dd1e88a26f3428086fae01488668589c1: Status 404 returned error can't find the container with id ff6780f67ba639c0b28b4aa1847ed13dd1e88a26f3428086fae01488668589c1 Oct 30 00:31:36.613566 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:36.613079 5001 generic.go:356] "Generic (PLEG): container finished" podID="a5ff3159-b9e7-4865-b35a-c2051d7e5c20" containerID="858956fd8fd665f36b2412e601621014cccbdbd473f21c4f2fdf5ce73b7a894c" exitCode=0 Oct 30 00:31:36.613566 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:36.613407 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" event={"ID":"a5ff3159-b9e7-4865-b35a-c2051d7e5c20","Type":"ContainerDied","Data":"858956fd8fd665f36b2412e601621014cccbdbd473f21c4f2fdf5ce73b7a894c"} Oct 30 00:31:36.653112 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:36.650339 5001 generic.go:356] "Generic (PLEG): container finished" podID="eda1038b-1cc5-4a86-934c-7c56e31be84b" containerID="9427d368df6f5ffd1d0a2f8be9f4336d79060fb44f7baeedb7259ffc70e4ee89" exitCode=0 Oct 30 00:31:36.653112 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:36.650547 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f649b0-4pgxm" event={"ID":"eda1038b-1cc5-4a86-934c-7c56e31be84b","Type":"ContainerDied","Data":"9427d368df6f5ffd1d0a2f8be9f4336d79060fb44f7baeedb7259ffc70e4ee89"} Oct 30 00:31:36.669295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:36.669254 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26df62a-8vccg" event={"ID":"405f5b62-eb46-429f-9f68-9f444812c478","Type":"ContainerStarted","Data":"ff6780f67ba639c0b28b4aa1847ed13dd1e88a26f3428086fae01488668589c1"} Oct 30 00:31:37.001766 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.001735 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:31:37.037839 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.037816 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:31:37.118090 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.118059 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/922f38ba-e017-4005-9841-aa0fd46b1d3c-kube-api-access-kdnl5\" (UniqueName: \"kubernetes.io/projected/922f38ba-e017-4005-9841-aa0fd46b1d3c-kube-api-access-kdnl5\") pod \"922f38ba-e017-4005-9841-aa0fd46b1d3c\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " Oct 30 00:31:37.118573 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.118126 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-istio-envoy\") pod \"922f38ba-e017-4005-9841-aa0fd46b1d3c\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " Oct 30 00:31:37.118573 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.118169 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istio-podinfo\") pod \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " Oct 30 00:31:37.118573 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.118194 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istio-data\") pod \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " Oct 30 00:31:37.118573 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.118223 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-workload-certs\") pod \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " Oct 30 00:31:37.118573 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.118248 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-credential-socket\") pod \"922f38ba-e017-4005-9841-aa0fd46b1d3c\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " Oct 30 00:31:37.118573 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.118278 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istio-envoy\") pod \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " Oct 30 00:31:37.118573 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.118306 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-istio-data\") pod \"922f38ba-e017-4005-9841-aa0fd46b1d3c\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " Oct 30 00:31:37.118573 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.118339 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-credential-socket\") pod \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " Oct 30 00:31:37.118573 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.118380 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istio-token\" (UniqueName: \"kubernetes.io/projected/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istio-token\") pod \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " Oct 30 00:31:37.118573 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.118407 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-workload-socket\") pod \"922f38ba-e017-4005-9841-aa0fd46b1d3c\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " Oct 30 00:31:37.118573 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.118520 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/922f38ba-e017-4005-9841-aa0fd46b1d3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922f38ba-e017-4005-9841-aa0fd46b1d3c-istiod-ca-cert\") pod \"922f38ba-e017-4005-9841-aa0fd46b1d3c\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " Oct 30 00:31:37.118573 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.118549 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-workload-certs\") pod \"922f38ba-e017-4005-9841-aa0fd46b1d3c\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " Oct 30 00:31:37.118573 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.118574 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/922f38ba-e017-4005-9841-aa0fd46b1d3c-istio-token\" (UniqueName: \"kubernetes.io/projected/922f38ba-e017-4005-9841-aa0fd46b1d3c-istio-token\") pod \"922f38ba-e017-4005-9841-aa0fd46b1d3c\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " Oct 30 00:31:37.119381 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.118667 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-workload-socket\") pod \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " Oct 30 00:31:37.119381 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.118719 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istiod-ca-cert\") pod \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " Oct 30 00:31:37.119381 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.118752 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-kube-api-access-dnpzj\" (UniqueName: \"kubernetes.io/projected/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-kube-api-access-dnpzj\") pod \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\" (UID: \"a5ff3159-b9e7-4865-b35a-c2051d7e5c20\") " Oct 30 00:31:37.119381 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.118805 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/922f38ba-e017-4005-9841-aa0fd46b1d3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922f38ba-e017-4005-9841-aa0fd46b1d3c-istio-podinfo\") pod \"922f38ba-e017-4005-9841-aa0fd46b1d3c\" (UID: \"922f38ba-e017-4005-9841-aa0fd46b1d3c\") " Oct 30 00:31:37.119381 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.118876 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-istio-data" pod "922f38ba-e017-4005-9841-aa0fd46b1d3c" (UID: "922f38ba-e017-4005-9841-aa0fd46b1d3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:37.119665 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.119600 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-workload-certs" pod "922f38ba-e017-4005-9841-aa0fd46b1d3c" (UID: "922f38ba-e017-4005-9841-aa0fd46b1d3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:37.119731 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.119692 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:37.119844 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.119821 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-credential-socket" pod "a5ff3159-b9e7-4865-b35a-c2051d7e5c20" (UID: "a5ff3159-b9e7-4865-b35a-c2051d7e5c20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:37.120942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.120452 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istiod-ca-cert" pod "a5ff3159-b9e7-4865-b35a-c2051d7e5c20" (UID: "a5ff3159-b9e7-4865-b35a-c2051d7e5c20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:31:37.120942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.120701 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-workload-socket" pod "a5ff3159-b9e7-4865-b35a-c2051d7e5c20" (UID: "a5ff3159-b9e7-4865-b35a-c2051d7e5c20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:37.122501 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.121656 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-istio-envoy" pod "922f38ba-e017-4005-9841-aa0fd46b1d3c" (UID: "922f38ba-e017-4005-9841-aa0fd46b1d3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:37.122501 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.122060 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-workload-certs" pod "a5ff3159-b9e7-4865-b35a-c2051d7e5c20" (UID: "a5ff3159-b9e7-4865-b35a-c2051d7e5c20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:37.122501 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.122292 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-workload-socket" pod "922f38ba-e017-4005-9841-aa0fd46b1d3c" (UID: "922f38ba-e017-4005-9841-aa0fd46b1d3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:37.122501 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.122333 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istio-data" pod "a5ff3159-b9e7-4865-b35a-c2051d7e5c20" (UID: "a5ff3159-b9e7-4865-b35a-c2051d7e5c20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:37.122501 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.122416 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istio-token" pod "a5ff3159-b9e7-4865-b35a-c2051d7e5c20" (UID: "a5ff3159-b9e7-4865-b35a-c2051d7e5c20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:31:37.122847 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.122617 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/922f38ba-e017-4005-9841-aa0fd46b1d3c-istiod-ca-cert" pod "922f38ba-e017-4005-9841-aa0fd46b1d3c" (UID: "922f38ba-e017-4005-9841-aa0fd46b1d3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:31:37.123123 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.123093 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-credential-socket" pod "922f38ba-e017-4005-9841-aa0fd46b1d3c" (UID: "922f38ba-e017-4005-9841-aa0fd46b1d3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:37.123679 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.123484 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb"] Oct 30 00:31:37.124184 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.124157 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istio-podinfo" pod "a5ff3159-b9e7-4865-b35a-c2051d7e5c20" (UID: "a5ff3159-b9e7-4865-b35a-c2051d7e5c20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:31:37.124419 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.124394 5001 cpu_manager.go:414] "containerMap: removing container" podUID="a5ff3159-b9e7-4865-b35a-c2051d7e5c20" containerName="istio-proxy" Oct 30 00:31:37.124419 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.124418 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5ff3159-b9e7-4865-b35a-c2051d7e5c20" containerName="istio-proxy" Oct 30 00:31:37.124586 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.124461 5001 cpu_manager.go:414] "containerMap: removing container" podUID="922f38ba-e017-4005-9841-aa0fd46b1d3c" containerName="istio-proxy" Oct 30 00:31:37.124586 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.124470 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="922f38ba-e017-4005-9841-aa0fd46b1d3c" containerName="istio-proxy" Oct 30 00:31:37.124726 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.124696 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5ff3159-b9e7-4865-b35a-c2051d7e5c20" containerName="istio-proxy" Oct 30 00:31:37.124726 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.124720 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="922f38ba-e017-4005-9841-aa0fd46b1d3c" containerName="istio-proxy" Oct 30 00:31:37.125009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.124981 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istio-envoy" pod "a5ff3159-b9e7-4865-b35a-c2051d7e5c20" (UID: "a5ff3159-b9e7-4865-b35a-c2051d7e5c20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:37.125479 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.125347 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922f38ba-e017-4005-9841-aa0fd46b1d3c-istio-token" pod "922f38ba-e017-4005-9841-aa0fd46b1d3c" (UID: "922f38ba-e017-4005-9841-aa0fd46b1d3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:31:37.125479 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.125456 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922f38ba-e017-4005-9841-aa0fd46b1d3c-kube-api-access-kdnl5" pod "922f38ba-e017-4005-9841-aa0fd46b1d3c" (UID: "922f38ba-e017-4005-9841-aa0fd46b1d3c"). InnerVolumeSpecName "kube-api-access-kdnl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:31:37.125822 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.125600 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-kube-api-access-dnpzj" pod "a5ff3159-b9e7-4865-b35a-c2051d7e5c20" (UID: "a5ff3159-b9e7-4865-b35a-c2051d7e5c20"). InnerVolumeSpecName "kube-api-access-dnpzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:31:37.127365 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.127341 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/922f38ba-e017-4005-9841-aa0fd46b1d3c-istio-podinfo" pod "922f38ba-e017-4005-9841-aa0fd46b1d3c" (UID: "922f38ba-e017-4005-9841-aa0fd46b1d3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:31:37.131249 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.131229 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:31:37.153682 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.153659 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb"] Oct 30 00:31:37.221001 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.220959 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:31:37.221205 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.221024 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:31:37.221205 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.221140 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:31:37.221355 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.221330 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:31:37.221413 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.221377 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:31:37.221491 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.221473 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:31:37.221552 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.221509 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqnn7\" (UniqueName: \"kubernetes.io/projected/d5dc33b0-82db-44f6-90df-dabe7979e9e6-kube-api-access-qqnn7\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:31:37.221552 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.221537 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:31:37.221671 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.221597 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:31:37.221726 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.221689 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:37.221726 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.221703 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:37.221842 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.221741 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnpzj\" (UniqueName: \"kubernetes.io/projected/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-kube-api-access-dnpzj\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:37.221842 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.221759 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922f38ba-e017-4005-9841-aa0fd46b1d3c-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:37.221842 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.221776 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdnl5\" (UniqueName: \"kubernetes.io/projected/922f38ba-e017-4005-9841-aa0fd46b1d3c-kube-api-access-kdnl5\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:37.221842 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.221791 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:37.221842 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.221806 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:37.221842 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.221820 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:37.221842 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.221834 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:37.222202 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.221847 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:37.222202 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.221861 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:37.222202 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.221875 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:37.222202 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.221890 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ff3159-b9e7-4865-b35a-c2051d7e5c20-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:37.222202 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.221904 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:37.222202 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.221955 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922f38ba-e017-4005-9841-aa0fd46b1d3c-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:37.222202 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.221969 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922f38ba-e017-4005-9841-aa0fd46b1d3c-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:37.222202 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.221981 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922f38ba-e017-4005-9841-aa0fd46b1d3c-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:37.323583 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.323549 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:31:37.323769 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.323602 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqnn7\" (UniqueName: \"kubernetes.io/projected/d5dc33b0-82db-44f6-90df-dabe7979e9e6-kube-api-access-qqnn7\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:31:37.323769 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.323635 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:31:37.323769 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.323664 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:31:37.323987 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.323785 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:31:37.323987 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.323846 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:31:37.323987 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.323881 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:31:37.324219 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.324033 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:31:37.324219 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.324116 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:31:37.324219 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.324122 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:31:37.324219 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.324182 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:31:37.324465 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.324420 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:31:37.324528 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.324495 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:31:37.324622 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.324599 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:31:37.325991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.325967 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:31:37.326391 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.326362 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:31:37.344111 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.344090 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:31:37.344325 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.344306 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqnn7\" (UniqueName: \"kubernetes.io/projected/d5dc33b0-82db-44f6-90df-dabe7979e9e6-kube-api-access-qqnn7\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:31:37.447361 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.447283 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:31:37.680833 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.680524 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26df62a-8vccg" event={"ID":"405f5b62-eb46-429f-9f68-9f444812c478","Type":"ContainerStarted","Data":"29c030eac9a6780fbf2e0ce66af394545f18630b614c59a925ff9b03e464a9f3"} Oct 30 00:31:37.682367 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.682346 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" Oct 30 00:31:37.682491 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.682394 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd" event={"ID":"a5ff3159-b9e7-4865-b35a-c2051d7e5c20","Type":"ContainerDied","Data":"0a6d036aec85bc7d5adf83f6dc4863fbbd50328cb475235c65c6c3b57d43eeb0"} Oct 30 00:31:37.682491 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.682483 5001 scope.go:122] "RemoveContainer" containerID="858956fd8fd665f36b2412e601621014cccbdbd473f21c4f2fdf5ce73b7a894c" Oct 30 00:31:37.684268 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.684244 5001 generic.go:356] "Generic (PLEG): container finished" podID="922f38ba-e017-4005-9841-aa0fd46b1d3c" containerID="b7fafa059445eb313e1005939bca2264a2ff067f01055e27dd2dc3fd1538a6a5" exitCode=0 Oct 30 00:31:37.684372 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.684300 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" Oct 30 00:31:37.684372 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.684328 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" event={"ID":"922f38ba-e017-4005-9841-aa0fd46b1d3c","Type":"ContainerDied","Data":"b7fafa059445eb313e1005939bca2264a2ff067f01055e27dd2dc3fd1538a6a5"} Oct 30 00:31:37.684495 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.684372 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm" event={"ID":"922f38ba-e017-4005-9841-aa0fd46b1d3c","Type":"ContainerDied","Data":"c8ce83852e80b2965abd13171c9f79b3d873805db3c301755d08002cb4aa25c7"} Oct 30 00:31:37.700740 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.700719 5001 scope.go:122] "RemoveContainer" containerID="b7fafa059445eb313e1005939bca2264a2ff067f01055e27dd2dc3fd1538a6a5" Oct 30 00:31:37.713390 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.713364 5001 scope.go:122] "RemoveContainer" containerID="b7fafa059445eb313e1005939bca2264a2ff067f01055e27dd2dc3fd1538a6a5" Oct 30 00:31:37.713826 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:31:37.713794 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7fafa059445eb313e1005939bca2264a2ff067f01055e27dd2dc3fd1538a6a5\": container with ID starting with b7fafa059445eb313e1005939bca2264a2ff067f01055e27dd2dc3fd1538a6a5 not found: ID does not exist" containerID="b7fafa059445eb313e1005939bca2264a2ff067f01055e27dd2dc3fd1538a6a5" Oct 30 00:31:37.713960 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.713848 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7fafa059445eb313e1005939bca2264a2ff067f01055e27dd2dc3fd1538a6a5"} err="failed to get container status \"b7fafa059445eb313e1005939bca2264a2ff067f01055e27dd2dc3fd1538a6a5\": rpc error: code = NotFound desc = could not find container \"b7fafa059445eb313e1005939bca2264a2ff067f01055e27dd2dc3fd1538a6a5\": container with ID starting with b7fafa059445eb313e1005939bca2264a2ff067f01055e27dd2dc3fd1538a6a5 not found: ID does not exist" Oct 30 00:31:37.719280 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.718863 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26df62a-8vccg" podStartSLOduration=2.718843055 podStartE2EDuration="2.718843055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:37.715635576 +0000 UTC m=+1612.921136757" watchObservedRunningTime="2026-10-30 00:31:37.718843055 +0000 UTC m=+1612.924344238" Oct 30 00:31:37.754963 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.751243 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd"] Oct 30 00:31:37.759496 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.758989 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-5m8kd"] Oct 30 00:31:37.776066 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.776044 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb"] Oct 30 00:31:37.781391 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.779786 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:31:37.781391 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.779855 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:31:37.781391 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.779885 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:31:37.786386 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.786365 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm"] Oct 30 00:31:37.793219 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.793199 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-j94gm"] Oct 30 00:31:37.948828 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:37.948804 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f649b0-4pgxm" Oct 30 00:31:38.038801 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:38.036190 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eda1038b-1cc5-4a86-934c-7c56e31be84b-kube-api-access-sdv8s\" (UniqueName: \"kubernetes.io/projected/eda1038b-1cc5-4a86-934c-7c56e31be84b-kube-api-access-sdv8s\") pod \"eda1038b-1cc5-4a86-934c-7c56e31be84b\" (UID: \"eda1038b-1cc5-4a86-934c-7c56e31be84b\") " Oct 30 00:31:38.038801 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:38.036301 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eda1038b-1cc5-4a86-934c-7c56e31be84b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eda1038b-1cc5-4a86-934c-7c56e31be84b-maas-api-openshift-ca\") pod \"eda1038b-1cc5-4a86-934c-7c56e31be84b\" (UID: \"eda1038b-1cc5-4a86-934c-7c56e31be84b\") " Oct 30 00:31:38.038801 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:38.036625 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eda1038b-1cc5-4a86-934c-7c56e31be84b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eda1038b-1cc5-4a86-934c-7c56e31be84b-maas-api-xks-ca\") pod \"eda1038b-1cc5-4a86-934c-7c56e31be84b\" (UID: \"eda1038b-1cc5-4a86-934c-7c56e31be84b\") " Oct 30 00:31:38.038801 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:38.037616 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eda1038b-1cc5-4a86-934c-7c56e31be84b-maas-api-openshift-ca" pod "eda1038b-1cc5-4a86-934c-7c56e31be84b" (UID: "eda1038b-1cc5-4a86-934c-7c56e31be84b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:31:38.043117 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:38.043055 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eda1038b-1cc5-4a86-934c-7c56e31be84b-kube-api-access-sdv8s" pod "eda1038b-1cc5-4a86-934c-7c56e31be84b" (UID: "eda1038b-1cc5-4a86-934c-7c56e31be84b"). InnerVolumeSpecName "kube-api-access-sdv8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:31:38.051546 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:38.051517 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eda1038b-1cc5-4a86-934c-7c56e31be84b-maas-api-xks-ca" pod "eda1038b-1cc5-4a86-934c-7c56e31be84b" (UID: "eda1038b-1cc5-4a86-934c-7c56e31be84b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:31:38.139342 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:38.139310 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eda1038b-1cc5-4a86-934c-7c56e31be84b-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:38.139342 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:38.139339 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eda1038b-1cc5-4a86-934c-7c56e31be84b-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:38.139342 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:38.139349 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdv8s\" (UniqueName: \"kubernetes.io/projected/eda1038b-1cc5-4a86-934c-7c56e31be84b-kube-api-access-sdv8s\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:38.694976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:38.694899 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" event={"ID":"d5dc33b0-82db-44f6-90df-dabe7979e9e6","Type":"ContainerStarted","Data":"2e6eb98fb28b392bbbf8c3617750e2bb799a1c9e87178a45effe71afa0e6cc64"} Oct 30 00:31:38.695158 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:38.695010 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" event={"ID":"d5dc33b0-82db-44f6-90df-dabe7979e9e6","Type":"ContainerStarted","Data":"64d910a1ccfd17fb994d9009af1245ef3061be2de453b7313391bc352f048e42"} Oct 30 00:31:38.696477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:38.696455 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f649b0-4pgxm" Oct 30 00:31:38.696627 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:38.696495 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f649b0-4pgxm" event={"ID":"eda1038b-1cc5-4a86-934c-7c56e31be84b","Type":"ContainerDied","Data":"6d54995e0573da9839f0056bc5602ba87a20e3cbfe1a57deec9a7da78999c68f"} Oct 30 00:31:38.696627 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:38.696539 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d54995e0573da9839f0056bc5602ba87a20e3cbfe1a57deec9a7da78999c68f" Oct 30 00:31:38.698121 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:38.698099 5001 generic.go:356] "Generic (PLEG): container finished" podID="405f5b62-eb46-429f-9f68-9f444812c478" containerID="29c030eac9a6780fbf2e0ce66af394545f18630b614c59a925ff9b03e464a9f3" exitCode=0 Oct 30 00:31:38.698289 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:38.698194 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26df62a-8vccg" event={"ID":"405f5b62-eb46-429f-9f68-9f444812c478","Type":"ContainerDied","Data":"29c030eac9a6780fbf2e0ce66af394545f18630b614c59a925ff9b03e464a9f3"} Oct 30 00:31:38.738550 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:38.738508 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" podStartSLOduration=1.738494489 podStartE2EDuration="1.738494489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:31:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:38.733568179 +0000 UTC m=+1613.939069364" watchObservedRunningTime="2026-10-30 00:31:38.738494489 +0000 UTC m=+1613.943995670" Oct 30 00:31:38.933061 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:38.932997 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="922f38ba-e017-4005-9841-aa0fd46b1d3c" path="/var/lib/kubelet/pods/922f38ba-e017-4005-9841-aa0fd46b1d3c/volumes" Oct 30 00:31:38.933497 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:38.933483 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5ff3159-b9e7-4865-b35a-c2051d7e5c20" path="/var/lib/kubelet/pods/a5ff3159-b9e7-4865-b35a-c2051d7e5c20/volumes" Oct 30 00:31:39.200076 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.200051 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:31:39.366969 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.366894 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-credential-socket\") pod \"44db3a17-452a-4667-bbc8-0f73b1711233\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " Oct 30 00:31:39.367135 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.366985 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-workload-socket\") pod \"44db3a17-452a-4667-bbc8-0f73b1711233\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " Oct 30 00:31:39.367135 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.367035 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-istio-data\") pod \"44db3a17-452a-4667-bbc8-0f73b1711233\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " Oct 30 00:31:39.367135 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.367086 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44db3a17-452a-4667-bbc8-0f73b1711233-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44db3a17-452a-4667-bbc8-0f73b1711233-istiod-ca-cert\") pod \"44db3a17-452a-4667-bbc8-0f73b1711233\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " Oct 30 00:31:39.367135 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.367130 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44db3a17-452a-4667-bbc8-0f73b1711233-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44db3a17-452a-4667-bbc8-0f73b1711233-istio-podinfo\") pod \"44db3a17-452a-4667-bbc8-0f73b1711233\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " Oct 30 00:31:39.367293 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.367187 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-workload-certs\") pod \"44db3a17-452a-4667-bbc8-0f73b1711233\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " Oct 30 00:31:39.367293 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.367198 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-credential-socket" pod "44db3a17-452a-4667-bbc8-0f73b1711233" (UID: "44db3a17-452a-4667-bbc8-0f73b1711233"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:39.367293 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.367254 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-istio-envoy\") pod \"44db3a17-452a-4667-bbc8-0f73b1711233\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " Oct 30 00:31:39.367421 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.367308 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44db3a17-452a-4667-bbc8-0f73b1711233-kube-api-access-jt9kx\" (UniqueName: \"kubernetes.io/projected/44db3a17-452a-4667-bbc8-0f73b1711233-kube-api-access-jt9kx\") pod \"44db3a17-452a-4667-bbc8-0f73b1711233\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " Oct 30 00:31:39.367421 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.367329 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-workload-socket" pod "44db3a17-452a-4667-bbc8-0f73b1711233" (UID: "44db3a17-452a-4667-bbc8-0f73b1711233"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:39.367421 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.367370 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44db3a17-452a-4667-bbc8-0f73b1711233-istio-token\" (UniqueName: \"kubernetes.io/projected/44db3a17-452a-4667-bbc8-0f73b1711233-istio-token\") pod \"44db3a17-452a-4667-bbc8-0f73b1711233\" (UID: \"44db3a17-452a-4667-bbc8-0f73b1711233\") " Oct 30 00:31:39.367421 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.367383 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-istio-data" pod "44db3a17-452a-4667-bbc8-0f73b1711233" (UID: "44db3a17-452a-4667-bbc8-0f73b1711233"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:39.367646 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.367516 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-workload-certs" pod "44db3a17-452a-4667-bbc8-0f73b1711233" (UID: "44db3a17-452a-4667-bbc8-0f73b1711233"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:39.367646 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.367572 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44db3a17-452a-4667-bbc8-0f73b1711233-istiod-ca-cert" pod "44db3a17-452a-4667-bbc8-0f73b1711233" (UID: "44db3a17-452a-4667-bbc8-0f73b1711233"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:31:39.368133 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.368107 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:39.368133 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.368134 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:39.368290 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.368144 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:39.368290 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.368153 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44db3a17-452a-4667-bbc8-0f73b1711233-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:39.368290 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.368161 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:39.369437 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.369413 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44db3a17-452a-4667-bbc8-0f73b1711233-istio-podinfo" pod "44db3a17-452a-4667-bbc8-0f73b1711233" (UID: "44db3a17-452a-4667-bbc8-0f73b1711233"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:31:39.369715 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.369691 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44db3a17-452a-4667-bbc8-0f73b1711233-kube-api-access-jt9kx" pod "44db3a17-452a-4667-bbc8-0f73b1711233" (UID: "44db3a17-452a-4667-bbc8-0f73b1711233"). InnerVolumeSpecName "kube-api-access-jt9kx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:31:39.369964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.369943 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-istio-envoy" pod "44db3a17-452a-4667-bbc8-0f73b1711233" (UID: "44db3a17-452a-4667-bbc8-0f73b1711233"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:31:39.369964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.369955 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44db3a17-452a-4667-bbc8-0f73b1711233-istio-token" pod "44db3a17-452a-4667-bbc8-0f73b1711233" (UID: "44db3a17-452a-4667-bbc8-0f73b1711233"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:31:39.448171 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.448079 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:31:39.450106 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.450081 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 00:31:39.450196 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.450127 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" podUID="d5dc33b0-82db-44f6-90df-dabe7979e9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 00:31:39.469686 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.469660 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44db3a17-452a-4667-bbc8-0f73b1711233-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:39.469686 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.469686 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44db3a17-452a-4667-bbc8-0f73b1711233-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:39.469814 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.469697 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt9kx\" (UniqueName: \"kubernetes.io/projected/44db3a17-452a-4667-bbc8-0f73b1711233-kube-api-access-jt9kx\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:39.469814 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.469709 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44db3a17-452a-4667-bbc8-0f73b1711233-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:39.737708 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.737612 5001 generic.go:356] "Generic (PLEG): container finished" podID="44db3a17-452a-4667-bbc8-0f73b1711233" containerID="a67316a0d7da125cb9b9eaee0e3e791f08ebc0afda6326926f7a9bd19799a8c5" exitCode=0 Oct 30 00:31:39.737708 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.737668 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" Oct 30 00:31:39.737989 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.737700 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" event={"ID":"44db3a17-452a-4667-bbc8-0f73b1711233","Type":"ContainerDied","Data":"a67316a0d7da125cb9b9eaee0e3e791f08ebc0afda6326926f7a9bd19799a8c5"} Oct 30 00:31:39.737989 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.737764 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm" event={"ID":"44db3a17-452a-4667-bbc8-0f73b1711233","Type":"ContainerDied","Data":"7d4dbc22cee988eda1103923780ee0d88506b4c9ae09e089d970037f77a6fbfd"} Oct 30 00:31:39.737989 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.737794 5001 scope.go:122] "RemoveContainer" containerID="a67316a0d7da125cb9b9eaee0e3e791f08ebc0afda6326926f7a9bd19799a8c5" Oct 30 00:31:39.751214 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.751193 5001 scope.go:122] "RemoveContainer" containerID="a67316a0d7da125cb9b9eaee0e3e791f08ebc0afda6326926f7a9bd19799a8c5" Oct 30 00:31:39.751548 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:31:39.751524 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a67316a0d7da125cb9b9eaee0e3e791f08ebc0afda6326926f7a9bd19799a8c5\": container with ID starting with a67316a0d7da125cb9b9eaee0e3e791f08ebc0afda6326926f7a9bd19799a8c5 not found: ID does not exist" containerID="a67316a0d7da125cb9b9eaee0e3e791f08ebc0afda6326926f7a9bd19799a8c5" Oct 30 00:31:39.751612 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.751560 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a67316a0d7da125cb9b9eaee0e3e791f08ebc0afda6326926f7a9bd19799a8c5"} err="failed to get container status \"a67316a0d7da125cb9b9eaee0e3e791f08ebc0afda6326926f7a9bd19799a8c5\": rpc error: code = NotFound desc = could not find container \"a67316a0d7da125cb9b9eaee0e3e791f08ebc0afda6326926f7a9bd19799a8c5\": container with ID starting with a67316a0d7da125cb9b9eaee0e3e791f08ebc0afda6326926f7a9bd19799a8c5 not found: ID does not exist" Oct 30 00:31:39.785595 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.785564 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm"] Oct 30 00:31:39.795867 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.795841 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fb3ad-openshift-default-58486b944f-wnsmm"] Oct 30 00:31:39.945972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:39.945946 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26df62a-8vccg" Oct 30 00:31:40.078995 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:40.078944 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/405f5b62-eb46-429f-9f68-9f444812c478-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405f5b62-eb46-429f-9f68-9f444812c478-maas-api-xks-ca\") pod \"405f5b62-eb46-429f-9f68-9f444812c478\" (UID: \"405f5b62-eb46-429f-9f68-9f444812c478\") " Oct 30 00:31:40.079200 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:40.079144 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/405f5b62-eb46-429f-9f68-9f444812c478-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405f5b62-eb46-429f-9f68-9f444812c478-maas-api-openshift-ca\") pod \"405f5b62-eb46-429f-9f68-9f444812c478\" (UID: \"405f5b62-eb46-429f-9f68-9f444812c478\") " Oct 30 00:31:40.079333 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:40.079317 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/405f5b62-eb46-429f-9f68-9f444812c478-kube-api-access-g8wdr\" (UniqueName: \"kubernetes.io/projected/405f5b62-eb46-429f-9f68-9f444812c478-kube-api-access-g8wdr\") pod \"405f5b62-eb46-429f-9f68-9f444812c478\" (UID: \"405f5b62-eb46-429f-9f68-9f444812c478\") " Oct 30 00:31:40.079569 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:40.079544 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/405f5b62-eb46-429f-9f68-9f444812c478-maas-api-openshift-ca" pod "405f5b62-eb46-429f-9f68-9f444812c478" (UID: "405f5b62-eb46-429f-9f68-9f444812c478"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:31:40.080140 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:40.079920 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405f5b62-eb46-429f-9f68-9f444812c478-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:40.081420 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:40.081398 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405f5b62-eb46-429f-9f68-9f444812c478-kube-api-access-g8wdr" pod "405f5b62-eb46-429f-9f68-9f444812c478" (UID: "405f5b62-eb46-429f-9f68-9f444812c478"). InnerVolumeSpecName "kube-api-access-g8wdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:31:40.081580 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:40.081563 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/405f5b62-eb46-429f-9f68-9f444812c478-maas-api-xks-ca" pod "405f5b62-eb46-429f-9f68-9f444812c478" (UID: "405f5b62-eb46-429f-9f68-9f444812c478"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:31:40.180976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:40.180899 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8wdr\" (UniqueName: \"kubernetes.io/projected/405f5b62-eb46-429f-9f68-9f444812c478-kube-api-access-g8wdr\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:40.180976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:40.180972 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405f5b62-eb46-429f-9f68-9f444812c478-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:40.448957 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:40.448458 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 00:31:40.448957 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:40.448513 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" podUID="d5dc33b0-82db-44f6-90df-dabe7979e9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 00:31:40.749094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:40.749004 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26df62a-8vccg" Oct 30 00:31:40.749094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:40.749027 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26df62a-8vccg" event={"ID":"405f5b62-eb46-429f-9f68-9f444812c478","Type":"ContainerDied","Data":"ff6780f67ba639c0b28b4aa1847ed13dd1e88a26f3428086fae01488668589c1"} Oct 30 00:31:40.749094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:40.749071 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff6780f67ba639c0b28b4aa1847ed13dd1e88a26f3428086fae01488668589c1" Oct 30 00:31:40.940728 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:40.940694 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44db3a17-452a-4667-bbc8-0f73b1711233" path="/var/lib/kubelet/pods/44db3a17-452a-4667-bbc8-0f73b1711233/volumes" Oct 30 00:31:41.447885 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:41.447851 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 00:31:41.448097 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:41.447902 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" podUID="d5dc33b0-82db-44f6-90df-dabe7979e9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 00:31:42.448012 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:42.447816 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 00:31:42.448012 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:42.447887 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" podUID="d5dc33b0-82db-44f6-90df-dabe7979e9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 00:31:43.160170 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:43.159033 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1dd8c1-x8z46"] Oct 30 00:31:43.183937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:43.182146 5001 cpu_manager.go:414] "containerMap: removing container" podUID="44db3a17-452a-4667-bbc8-0f73b1711233" containerName="istio-proxy" Oct 30 00:31:43.183937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:43.182375 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44db3a17-452a-4667-bbc8-0f73b1711233" containerName="istio-proxy" Oct 30 00:31:43.183937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:43.182433 5001 cpu_manager.go:414] "containerMap: removing container" podUID="405f5b62-eb46-429f-9f68-9f444812c478" containerName="revoke-keys" Oct 30 00:31:43.183937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:43.182444 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="405f5b62-eb46-429f-9f68-9f444812c478" containerName="revoke-keys" Oct 30 00:31:43.183937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:43.182661 5001 cpu_manager.go:414] "containerMap: removing container" podUID="eda1038b-1cc5-4a86-934c-7c56e31be84b" containerName="revoke-keys" Oct 30 00:31:43.183937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:43.182672 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eda1038b-1cc5-4a86-934c-7c56e31be84b" containerName="revoke-keys" Oct 30 00:31:43.189972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:43.184729 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="44db3a17-452a-4667-bbc8-0f73b1711233" containerName="istio-proxy" Oct 30 00:31:43.189972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:43.184757 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="405f5b62-eb46-429f-9f68-9f444812c478" containerName="revoke-keys" Oct 30 00:31:43.189972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:43.184770 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="eda1038b-1cc5-4a86-934c-7c56e31be84b" containerName="revoke-keys" Oct 30 00:31:43.265720 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:43.265683 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1dd8c1-x8z46"] Oct 30 00:31:43.265951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:43.265705 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1dd8c1-x8z46" Oct 30 00:31:43.278284 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:43.278261 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:31:43.288341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:43.288318 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:31:43.288341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:43.288318 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:31:43.288451 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:43.288324 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:31:43.356506 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:43.356469 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd26fcf0-a65b-40e8-ad15-213b90e866bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1dd8c1-x8z46\" (UID: \"cd26fcf0-a65b-40e8-ad15-213b90e866bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1dd8c1-x8z46" Oct 30 00:31:43.356676 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:43.356632 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd26fcf0-a65b-40e8-ad15-213b90e866bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1dd8c1-x8z46\" (UID: \"cd26fcf0-a65b-40e8-ad15-213b90e866bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1dd8c1-x8z46" Oct 30 00:31:43.356676 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:43.356663 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7d5b\" (UniqueName: \"kubernetes.io/projected/cd26fcf0-a65b-40e8-ad15-213b90e866bc-kube-api-access-w7d5b\") pod \"maas-api-revoke-keys-e2e-praxis-a-1dd8c1-x8z46\" (UID: \"cd26fcf0-a65b-40e8-ad15-213b90e866bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1dd8c1-x8z46" Oct 30 00:31:43.448292 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:43.448208 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 00:31:43.448292 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:43.448259 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" podUID="d5dc33b0-82db-44f6-90df-dabe7979e9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 00:31:43.458075 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:43.458039 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd26fcf0-a65b-40e8-ad15-213b90e866bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1dd8c1-x8z46\" (UID: \"cd26fcf0-a65b-40e8-ad15-213b90e866bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1dd8c1-x8z46" Oct 30 00:31:43.458165 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:43.458109 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7d5b\" (UniqueName: \"kubernetes.io/projected/cd26fcf0-a65b-40e8-ad15-213b90e866bc-kube-api-access-w7d5b\") pod \"maas-api-revoke-keys-e2e-praxis-a-1dd8c1-x8z46\" (UID: \"cd26fcf0-a65b-40e8-ad15-213b90e866bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1dd8c1-x8z46" Oct 30 00:31:43.458293 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:43.458269 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd26fcf0-a65b-40e8-ad15-213b90e866bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1dd8c1-x8z46\" (UID: \"cd26fcf0-a65b-40e8-ad15-213b90e866bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1dd8c1-x8z46" Oct 30 00:31:43.458935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:43.458891 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd26fcf0-a65b-40e8-ad15-213b90e866bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1dd8c1-x8z46\" (UID: \"cd26fcf0-a65b-40e8-ad15-213b90e866bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1dd8c1-x8z46" Oct 30 00:31:43.460496 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:43.460471 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd26fcf0-a65b-40e8-ad15-213b90e866bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1dd8c1-x8z46\" (UID: \"cd26fcf0-a65b-40e8-ad15-213b90e866bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1dd8c1-x8z46" Oct 30 00:31:43.478688 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:43.478664 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7d5b\" (UniqueName: \"kubernetes.io/projected/cd26fcf0-a65b-40e8-ad15-213b90e866bc-kube-api-access-w7d5b\") pod \"maas-api-revoke-keys-e2e-praxis-a-1dd8c1-x8z46\" (UID: \"cd26fcf0-a65b-40e8-ad15-213b90e866bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1dd8c1-x8z46" Oct 30 00:31:43.582069 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:43.582039 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1dd8c1-x8z46" Oct 30 00:31:43.955872 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:43.955837 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1dd8c1-x8z46"] Oct 30 00:31:44.448483 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:44.448444 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 00:31:44.448890 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:44.448512 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" podUID="d5dc33b0-82db-44f6-90df-dabe7979e9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 00:31:44.793289 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:44.793198 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1dd8c1-x8z46" event={"ID":"cd26fcf0-a65b-40e8-ad15-213b90e866bc","Type":"ContainerStarted","Data":"d3692f4a0d7e9fc815f4a2d7e41f1ee6b9df3d6d6f30a8671ef5c4b56cb5a6fc"} Oct 30 00:31:44.793289 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:44.793265 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1dd8c1-x8z46" event={"ID":"cd26fcf0-a65b-40e8-ad15-213b90e866bc","Type":"ContainerStarted","Data":"7dd212e68b53efcd503325095372f3e4d9c2dc309d4f388504200df8ba85d8e3"} Oct 30 00:31:44.833028 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:44.832965 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1dd8c1-x8z46" podStartSLOduration=1.832947155 podStartE2EDuration="1.832947155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:44.829416221 +0000 UTC m=+1620.034917402" watchObservedRunningTime="2026-10-30 00:31:44.832947155 +0000 UTC m=+1620.038448363" Oct 30 00:31:45.448477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:45.448443 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 00:31:45.448652 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:45.448508 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" podUID="d5dc33b0-82db-44f6-90df-dabe7979e9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 00:31:45.809243 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:45.809199 5001 generic.go:356] "Generic (PLEG): container finished" podID="cd26fcf0-a65b-40e8-ad15-213b90e866bc" containerID="d3692f4a0d7e9fc815f4a2d7e41f1ee6b9df3d6d6f30a8671ef5c4b56cb5a6fc" exitCode=0 Oct 30 00:31:45.809441 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:45.809271 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1dd8c1-x8z46" event={"ID":"cd26fcf0-a65b-40e8-ad15-213b90e866bc","Type":"ContainerDied","Data":"d3692f4a0d7e9fc815f4a2d7e41f1ee6b9df3d6d6f30a8671ef5c4b56cb5a6fc"} Oct 30 00:31:46.452057 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:46.452027 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:31:46.452578 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:46.452560 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:31:46.453220 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:46.453200 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:31:46.997872 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:46.997847 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1dd8c1-x8z46" Oct 30 00:31:47.017285 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:47.017257 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd26fcf0-a65b-40e8-ad15-213b90e866bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd26fcf0-a65b-40e8-ad15-213b90e866bc-maas-api-openshift-ca\") pod \"cd26fcf0-a65b-40e8-ad15-213b90e866bc\" (UID: \"cd26fcf0-a65b-40e8-ad15-213b90e866bc\") " Oct 30 00:31:47.017487 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:47.017470 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd26fcf0-a65b-40e8-ad15-213b90e866bc-kube-api-access-w7d5b\" (UniqueName: \"kubernetes.io/projected/cd26fcf0-a65b-40e8-ad15-213b90e866bc-kube-api-access-w7d5b\") pod \"cd26fcf0-a65b-40e8-ad15-213b90e866bc\" (UID: \"cd26fcf0-a65b-40e8-ad15-213b90e866bc\") " Oct 30 00:31:47.017556 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:47.017507 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd26fcf0-a65b-40e8-ad15-213b90e866bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd26fcf0-a65b-40e8-ad15-213b90e866bc-maas-api-xks-ca\") pod \"cd26fcf0-a65b-40e8-ad15-213b90e866bc\" (UID: \"cd26fcf0-a65b-40e8-ad15-213b90e866bc\") " Oct 30 00:31:47.017614 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:47.017566 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd26fcf0-a65b-40e8-ad15-213b90e866bc-maas-api-openshift-ca" pod "cd26fcf0-a65b-40e8-ad15-213b90e866bc" (UID: "cd26fcf0-a65b-40e8-ad15-213b90e866bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:31:47.018424 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:47.018149 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd26fcf0-a65b-40e8-ad15-213b90e866bc-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:47.019661 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:47.019635 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd26fcf0-a65b-40e8-ad15-213b90e866bc-maas-api-xks-ca" pod "cd26fcf0-a65b-40e8-ad15-213b90e866bc" (UID: "cd26fcf0-a65b-40e8-ad15-213b90e866bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:31:47.019723 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:47.019642 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd26fcf0-a65b-40e8-ad15-213b90e866bc-kube-api-access-w7d5b" pod "cd26fcf0-a65b-40e8-ad15-213b90e866bc" (UID: "cd26fcf0-a65b-40e8-ad15-213b90e866bc"). InnerVolumeSpecName "kube-api-access-w7d5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:31:47.124317 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:47.123846 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7d5b\" (UniqueName: \"kubernetes.io/projected/cd26fcf0-a65b-40e8-ad15-213b90e866bc-kube-api-access-w7d5b\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:47.124317 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:47.123885 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd26fcf0-a65b-40e8-ad15-213b90e866bc-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:47.331260 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:31:47.331220 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:31:47.831464 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:47.831420 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1dd8c1-x8z46" event={"ID":"cd26fcf0-a65b-40e8-ad15-213b90e866bc","Type":"ContainerDied","Data":"7dd212e68b53efcd503325095372f3e4d9c2dc309d4f388504200df8ba85d8e3"} Oct 30 00:31:47.831893 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:47.831484 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1dd8c1-x8z46" Oct 30 00:31:47.831893 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:47.831537 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7dd212e68b53efcd503325095372f3e4d9c2dc309d4f388504200df8ba85d8e3" Oct 30 00:31:50.087964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:50.084230 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3ea63d4-vmfh9"] Oct 30 00:31:50.087964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:50.085342 5001 cpu_manager.go:414] "containerMap: removing container" podUID="cd26fcf0-a65b-40e8-ad15-213b90e866bc" containerName="revoke-keys" Oct 30 00:31:50.087964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:50.085365 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd26fcf0-a65b-40e8-ad15-213b90e866bc" containerName="revoke-keys" Oct 30 00:31:50.087964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:50.085533 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd26fcf0-a65b-40e8-ad15-213b90e866bc" containerName="revoke-keys" Oct 30 00:31:50.090339 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:50.090314 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ea63d4-vmfh9" Oct 30 00:31:50.143955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:50.136225 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:31:50.347006 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:50.341316 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:31:50.579060 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:50.358803 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3ea63d4-vmfh9"] Oct 30 00:31:50.579404 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:50.358971 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:31:50.594844 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:50.484218 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:31:50.595079 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:50.578573 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9f1f3d-4112-4ce6-bff3-c789cc809901-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3ea63d4-vmfh9\" (UID: \"aa9f1f3d-4112-4ce6-bff3-c789cc809901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ea63d4-vmfh9" Oct 30 00:31:50.595362 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:50.595338 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9f1f3d-4112-4ce6-bff3-c789cc809901-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3ea63d4-vmfh9\" (UID: \"aa9f1f3d-4112-4ce6-bff3-c789cc809901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ea63d4-vmfh9" Oct 30 00:31:50.738990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:50.726241 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crzcq\" (UniqueName: \"kubernetes.io/projected/aa9f1f3d-4112-4ce6-bff3-c789cc809901-kube-api-access-crzcq\") pod \"maas-api-revoke-sub-f3ea63d4-vmfh9\" (UID: \"aa9f1f3d-4112-4ce6-bff3-c789cc809901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ea63d4-vmfh9" Oct 30 00:31:50.738990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:50.726549 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9f1f3d-4112-4ce6-bff3-c789cc809901-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3ea63d4-vmfh9\" (UID: \"aa9f1f3d-4112-4ce6-bff3-c789cc809901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ea63d4-vmfh9" Oct 30 00:31:50.738990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:50.726666 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9f1f3d-4112-4ce6-bff3-c789cc809901-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3ea63d4-vmfh9\" (UID: \"aa9f1f3d-4112-4ce6-bff3-c789cc809901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ea63d4-vmfh9" Oct 30 00:31:50.738990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:50.727738 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9f1f3d-4112-4ce6-bff3-c789cc809901-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3ea63d4-vmfh9\" (UID: \"aa9f1f3d-4112-4ce6-bff3-c789cc809901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ea63d4-vmfh9" Oct 30 00:31:50.772937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:50.768448 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9f1f3d-4112-4ce6-bff3-c789cc809901-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3ea63d4-vmfh9\" (UID: \"aa9f1f3d-4112-4ce6-bff3-c789cc809901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ea63d4-vmfh9" Oct 30 00:31:50.835389 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:50.835347 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crzcq\" (UniqueName: \"kubernetes.io/projected/aa9f1f3d-4112-4ce6-bff3-c789cc809901-kube-api-access-crzcq\") pod \"maas-api-revoke-sub-f3ea63d4-vmfh9\" (UID: \"aa9f1f3d-4112-4ce6-bff3-c789cc809901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ea63d4-vmfh9" Oct 30 00:31:50.862665 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:50.862631 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crzcq\" (UniqueName: \"kubernetes.io/projected/aa9f1f3d-4112-4ce6-bff3-c789cc809901-kube-api-access-crzcq\") pod \"maas-api-revoke-sub-f3ea63d4-vmfh9\" (UID: \"aa9f1f3d-4112-4ce6-bff3-c789cc809901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ea63d4-vmfh9" Oct 30 00:31:51.061206 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:51.061167 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ea63d4-vmfh9" Oct 30 00:31:51.890667 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:51.890625 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3ea63d4-vmfh9"] Oct 30 00:31:51.942858 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:51.942824 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ea63d4-vmfh9" event={"ID":"aa9f1f3d-4112-4ce6-bff3-c789cc809901","Type":"ContainerStarted","Data":"0109bebf41ea6ef2769239115b2424567222c3e3349fbe669afa21a8a51bc807"} Oct 30 00:31:52.953063 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:52.953028 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ea63d4-vmfh9" event={"ID":"aa9f1f3d-4112-4ce6-bff3-c789cc809901","Type":"ContainerStarted","Data":"2460abf827d74394c9b7e3a6c689d116c9d7f324513050df272e29ca8a6d577c"} Oct 30 00:31:52.990182 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:52.990127 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ea63d4-vmfh9" podStartSLOduration=2.990111186 podStartE2EDuration="2.990111186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:52.987701692 +0000 UTC m=+1628.193202874" watchObservedRunningTime="2026-10-30 00:31:52.990111186 +0000 UTC m=+1628.195612370" Oct 30 00:31:53.568059 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:53.568026 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1dd8c1-755bcc85b6-2zdss"] Oct 30 00:31:53.988206 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:53.988125 5001 generic.go:356] "Generic (PLEG): container finished" podID="aa9f1f3d-4112-4ce6-bff3-c789cc809901" containerID="2460abf827d74394c9b7e3a6c689d116c9d7f324513050df272e29ca8a6d577c" exitCode=0 Oct 30 00:31:53.988577 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:53.988238 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ea63d4-vmfh9" event={"ID":"aa9f1f3d-4112-4ce6-bff3-c789cc809901","Type":"ContainerDied","Data":"2460abf827d74394c9b7e3a6c689d116c9d7f324513050df272e29ca8a6d577c"} Oct 30 00:31:55.370717 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:55.370685 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ea63d4-vmfh9" Oct 30 00:31:55.494498 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:55.494463 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa9f1f3d-4112-4ce6-bff3-c789cc809901-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9f1f3d-4112-4ce6-bff3-c789cc809901-maas-api-openshift-ca\") pod \"aa9f1f3d-4112-4ce6-bff3-c789cc809901\" (UID: \"aa9f1f3d-4112-4ce6-bff3-c789cc809901\") " Oct 30 00:31:55.494693 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:55.494634 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa9f1f3d-4112-4ce6-bff3-c789cc809901-kube-api-access-crzcq\" (UniqueName: \"kubernetes.io/projected/aa9f1f3d-4112-4ce6-bff3-c789cc809901-kube-api-access-crzcq\") pod \"aa9f1f3d-4112-4ce6-bff3-c789cc809901\" (UID: \"aa9f1f3d-4112-4ce6-bff3-c789cc809901\") " Oct 30 00:31:55.494743 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:55.494698 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa9f1f3d-4112-4ce6-bff3-c789cc809901-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9f1f3d-4112-4ce6-bff3-c789cc809901-maas-api-xks-ca\") pod \"aa9f1f3d-4112-4ce6-bff3-c789cc809901\" (UID: \"aa9f1f3d-4112-4ce6-bff3-c789cc809901\") " Oct 30 00:31:55.495362 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:55.495328 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa9f1f3d-4112-4ce6-bff3-c789cc809901-maas-api-openshift-ca" pod "aa9f1f3d-4112-4ce6-bff3-c789cc809901" (UID: "aa9f1f3d-4112-4ce6-bff3-c789cc809901"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:31:55.495591 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:55.495566 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9f1f3d-4112-4ce6-bff3-c789cc809901-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:55.497216 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:55.497190 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa9f1f3d-4112-4ce6-bff3-c789cc809901-kube-api-access-crzcq" pod "aa9f1f3d-4112-4ce6-bff3-c789cc809901" (UID: "aa9f1f3d-4112-4ce6-bff3-c789cc809901"). InnerVolumeSpecName "kube-api-access-crzcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:31:55.497417 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:55.497399 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa9f1f3d-4112-4ce6-bff3-c789cc809901-maas-api-xks-ca" pod "aa9f1f3d-4112-4ce6-bff3-c789cc809901" (UID: "aa9f1f3d-4112-4ce6-bff3-c789cc809901"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:31:55.596767 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:55.596691 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crzcq\" (UniqueName: \"kubernetes.io/projected/aa9f1f3d-4112-4ce6-bff3-c789cc809901-kube-api-access-crzcq\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:55.596767 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:55.596721 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9f1f3d-4112-4ce6-bff3-c789cc809901-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:56.024933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:56.021970 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ea63d4-vmfh9" event={"ID":"aa9f1f3d-4112-4ce6-bff3-c789cc809901","Type":"ContainerDied","Data":"0109bebf41ea6ef2769239115b2424567222c3e3349fbe669afa21a8a51bc807"} Oct 30 00:31:56.024933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:56.022035 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0109bebf41ea6ef2769239115b2424567222c3e3349fbe669afa21a8a51bc807" Oct 30 00:31:56.024933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:56.022104 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ea63d4-vmfh9" Oct 30 00:31:57.389269 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:57.389225 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f649b0-57947cf4fd-qdghs"] Oct 30 00:31:57.389843 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:57.389489 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-57947cf4fd-qdghs" podUID="4925fd35-f8b1-4706-990d-a291bb1ed760" containerName="payload-processing" containerID="cri-o://86e07d664b557dcfb0583e48d74116053c99e34286cf00a934adf371ef49abcb" gracePeriod=30 Oct 30 00:31:57.433973 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:57.420601 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-75dd767db7-8fdnd"] Oct 30 00:31:57.433973 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:57.420841 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-75dd767db7-8fdnd" podUID="820af093-87be-4078-9098-1997a629c2c7" containerName="payload-pre-processing" containerID="cri-o://1c82807655b4491ce427e14febc1fd7b071eda9eb55763e2d74f3b6fa5cb024e" gracePeriod=30 Oct 30 00:31:57.959943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:57.919829 5001 scope.go:122] "RemoveContainer" containerID="951177bcf7f535391505c41b5321337cf90af5467d26ce382a588652cb91e60c" Oct 30 00:31:58.004378 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:57.975194 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f649b0-4pgxm"] Oct 30 00:31:58.004378 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:57.981672 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f649b0-4pgxm"] Oct 30 00:31:58.205181 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:58.205146 5001 generic.go:356] "Generic (PLEG): container finished" podID="820af093-87be-4078-9098-1997a629c2c7" containerID="1c82807655b4491ce427e14febc1fd7b071eda9eb55763e2d74f3b6fa5cb024e" exitCode=0 Oct 30 00:31:58.207117 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:58.207081 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-75dd767db7-8fdnd" event={"ID":"820af093-87be-4078-9098-1997a629c2c7","Type":"ContainerDied","Data":"1c82807655b4491ce427e14febc1fd7b071eda9eb55763e2d74f3b6fa5cb024e"} Oct 30 00:31:58.327147 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:58.326484 5001 scope.go:122] "RemoveContainer" containerID="674d726a595c86b83bfc3c06cbf202a45dbcb5c1018260a9bdbf049fb269c85a" Oct 30 00:31:58.381420 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:58.380230 5001 generic.go:356] "Generic (PLEG): container finished" podID="4925fd35-f8b1-4706-990d-a291bb1ed760" containerID="86e07d664b557dcfb0583e48d74116053c99e34286cf00a934adf371ef49abcb" exitCode=0 Oct 30 00:31:58.381420 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:58.380448 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-57947cf4fd-qdghs" event={"ID":"4925fd35-f8b1-4706-990d-a291bb1ed760","Type":"ContainerDied","Data":"86e07d664b557dcfb0583e48d74116053c99e34286cf00a934adf371ef49abcb"} Oct 30 00:31:58.569725 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:58.565607 5001 scope.go:122] "RemoveContainer" containerID="80cd54c2ee4c671480b0df1d9c2f1af5dae91887f808bec5a6f1e40f3b4e9211" Oct 30 00:31:58.569725 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:58.568549 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-1dd8c1-755bcc85b6-2zdss" podUID="65938fbc-9537-4aa2-b0b0-a77ff94384ca" containerName="payload-processing" containerID="cri-o://dddb0d1ab2dea99b9dc6eb5563c4f90eaa51c66c53f8f715b87feac27cc5d004" gracePeriod=25 Oct 30 00:31:59.136400 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.136358 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eda1038b-1cc5-4a86-934c-7c56e31be84b" path="/var/lib/kubelet/pods/eda1038b-1cc5-4a86-934c-7c56e31be84b/volumes" Oct 30 00:31:59.139141 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.139116 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn"] Oct 30 00:31:59.140542 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.140513 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" podUID="34da2ec4-9938-4eaf-9845-dc23bcbc761a" containerName="istio-proxy" containerID="cri-o://f10fe7ca539096970b05937017c10fc1e492f75a6174b548cc05fd9cfd2c712b" gracePeriod=30 Oct 30 00:31:59.143507 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.143487 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-75dd767db7-8fdnd" Oct 30 00:31:59.143974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.143955 5001 scope.go:122] "RemoveContainer" containerID="6a446e0894090fcc46a51bb6a1dc982d1af2dfcf9055f72ea27ab61e444f4abd" Oct 30 00:31:59.148639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.148610 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-57947cf4fd-qdghs" Oct 30 00:31:59.165158 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.165135 5001 scope.go:122] "RemoveContainer" containerID="40a9e950830a91bf6053e823c280f93823258336a2915ab28e56ee03dbf124d6" Oct 30 00:31:59.180130 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:31:59.180078 5001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65938fbc_9537_4aa2_b0b0_a77ff94384ca.slice/crio-dddb0d1ab2dea99b9dc6eb5563c4f90eaa51c66c53f8f715b87feac27cc5d004.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65938fbc_9537_4aa2_b0b0_a77ff94384ca.slice/crio-conmon-dddb0d1ab2dea99b9dc6eb5563c4f90eaa51c66c53f8f715b87feac27cc5d004.scope\": RecentStats: unable to find data in memory cache]" Oct 30 00:31:59.191342 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.189486 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/820af093-87be-4078-9098-1997a629c2c7-kube-api-access-dvlm4\" (UniqueName: \"kubernetes.io/projected/820af093-87be-4078-9098-1997a629c2c7-kube-api-access-dvlm4\") pod \"820af093-87be-4078-9098-1997a629c2c7\" (UID: \"820af093-87be-4078-9098-1997a629c2c7\") " Oct 30 00:31:59.191342 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.189642 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/820af093-87be-4078-9098-1997a629c2c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/820af093-87be-4078-9098-1997a629c2c7-plugins-config-volume\") pod \"820af093-87be-4078-9098-1997a629c2c7\" (UID: \"820af093-87be-4078-9098-1997a629c2c7\") " Oct 30 00:31:59.192083 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.192001 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/820af093-87be-4078-9098-1997a629c2c7-plugins-config-volume" pod "820af093-87be-4078-9098-1997a629c2c7" (UID: "820af093-87be-4078-9098-1997a629c2c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:31:59.194855 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.194828 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/820af093-87be-4078-9098-1997a629c2c7-kube-api-access-dvlm4" pod "820af093-87be-4078-9098-1997a629c2c7" (UID: "820af093-87be-4078-9098-1997a629c2c7"). InnerVolumeSpecName "kube-api-access-dvlm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:31:59.262581 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.257019 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3e13d-gptcf"] Oct 30 00:31:59.262581 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.258041 5001 cpu_manager.go:414] "containerMap: removing container" podUID="aa9f1f3d-4112-4ce6-bff3-c789cc809901" containerName="revoke-keys" Oct 30 00:31:59.262581 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.258061 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa9f1f3d-4112-4ce6-bff3-c789cc809901" containerName="revoke-keys" Oct 30 00:31:59.262581 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.258089 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4925fd35-f8b1-4706-990d-a291bb1ed760" containerName="payload-processing" Oct 30 00:31:59.262581 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.258096 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4925fd35-f8b1-4706-990d-a291bb1ed760" containerName="payload-processing" Oct 30 00:31:59.262581 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.258143 5001 cpu_manager.go:414] "containerMap: removing container" podUID="820af093-87be-4078-9098-1997a629c2c7" containerName="payload-pre-processing" Oct 30 00:31:59.262581 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.258152 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="820af093-87be-4078-9098-1997a629c2c7" containerName="payload-pre-processing" Oct 30 00:31:59.262581 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.258315 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa9f1f3d-4112-4ce6-bff3-c789cc809901" containerName="revoke-keys" Oct 30 00:31:59.262581 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.258328 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4925fd35-f8b1-4706-990d-a291bb1ed760" containerName="payload-processing" Oct 30 00:31:59.262581 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.258346 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="820af093-87be-4078-9098-1997a629c2c7" containerName="payload-pre-processing" Oct 30 00:31:59.266959 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.266897 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3e13d-gptcf" Oct 30 00:31:59.268897 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.268791 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3e13d-gptcf"] Oct 30 00:31:59.279955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.279930 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:31:59.280169 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.280150 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:31:59.280386 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.280363 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:31:59.280481 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.280452 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:31:59.290885 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.290857 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4925fd35-f8b1-4706-990d-a291bb1ed760-kube-api-access-mnjg5\" (UniqueName: \"kubernetes.io/projected/4925fd35-f8b1-4706-990d-a291bb1ed760-kube-api-access-mnjg5\") pod \"4925fd35-f8b1-4706-990d-a291bb1ed760\" (UID: \"4925fd35-f8b1-4706-990d-a291bb1ed760\") " Oct 30 00:31:59.296184 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.291061 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4925fd35-f8b1-4706-990d-a291bb1ed760-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4925fd35-f8b1-4706-990d-a291bb1ed760-plugins-config-volume\") pod \"4925fd35-f8b1-4706-990d-a291bb1ed760\" (UID: \"4925fd35-f8b1-4706-990d-a291bb1ed760\") " Oct 30 00:31:59.296184 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.291592 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvlm4\" (UniqueName: \"kubernetes.io/projected/820af093-87be-4078-9098-1997a629c2c7-kube-api-access-dvlm4\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:59.296184 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.291608 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/820af093-87be-4078-9098-1997a629c2c7-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:59.296184 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.291877 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4925fd35-f8b1-4706-990d-a291bb1ed760-plugins-config-volume" pod "4925fd35-f8b1-4706-990d-a291bb1ed760" (UID: "4925fd35-f8b1-4706-990d-a291bb1ed760"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:31:59.301656 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.301636 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4925fd35-f8b1-4706-990d-a291bb1ed760-kube-api-access-mnjg5" pod "4925fd35-f8b1-4706-990d-a291bb1ed760" (UID: "4925fd35-f8b1-4706-990d-a291bb1ed760"). InnerVolumeSpecName "kube-api-access-mnjg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:31:59.393152 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.392726 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-d3e13d-gptcf\" (UID: \"4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3e13d-gptcf" Oct 30 00:31:59.393152 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.392856 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvcrr\" (UniqueName: \"kubernetes.io/projected/4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c-kube-api-access-kvcrr\") pod \"maas-api-revoke-keys-e2e-praxis-b-d3e13d-gptcf\" (UID: \"4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3e13d-gptcf" Oct 30 00:31:59.393152 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.392941 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-d3e13d-gptcf\" (UID: \"4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3e13d-gptcf" Oct 30 00:31:59.393152 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.393047 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnjg5\" (UniqueName: \"kubernetes.io/projected/4925fd35-f8b1-4706-990d-a291bb1ed760-kube-api-access-mnjg5\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:59.393152 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.393064 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4925fd35-f8b1-4706-990d-a291bb1ed760-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:59.401098 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.400256 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-75dd767db7-8fdnd" event={"ID":"820af093-87be-4078-9098-1997a629c2c7","Type":"ContainerDied","Data":"4a678a32acc13c328af0460f8cdbbfee13014a940d1513216cc3450acc3bad96"} Oct 30 00:31:59.401098 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.400336 5001 scope.go:122] "RemoveContainer" containerID="1c82807655b4491ce427e14febc1fd7b071eda9eb55763e2d74f3b6fa5cb024e" Oct 30 00:31:59.401098 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.400504 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-75dd767db7-8fdnd" Oct 30 00:31:59.446588 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.446553 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-57947cf4fd-qdghs" Oct 30 00:31:59.448100 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.446550 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f649b0-57947cf4fd-qdghs" event={"ID":"4925fd35-f8b1-4706-990d-a291bb1ed760","Type":"ContainerDied","Data":"e045aa0493885d3bcdac506ce06ca3b05b507616885911db9fd3a70603f8f40f"} Oct 30 00:31:59.451143 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.451066 5001 generic.go:356] "Generic (PLEG): container finished" podID="65938fbc-9537-4aa2-b0b0-a77ff94384ca" containerID="dddb0d1ab2dea99b9dc6eb5563c4f90eaa51c66c53f8f715b87feac27cc5d004" exitCode=0 Oct 30 00:31:59.451488 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.451369 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1dd8c1-755bcc85b6-2zdss" event={"ID":"65938fbc-9537-4aa2-b0b0-a77ff94384ca","Type":"ContainerDied","Data":"dddb0d1ab2dea99b9dc6eb5563c4f90eaa51c66c53f8f715b87feac27cc5d004"} Oct 30 00:31:59.455629 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.455518 5001 scope.go:122] "RemoveContainer" containerID="86e07d664b557dcfb0583e48d74116053c99e34286cf00a934adf371ef49abcb" Oct 30 00:31:59.471113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.471095 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1dd8c1-755bcc85b6-2zdss" Oct 30 00:31:59.486016 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.485991 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-75dd767db7-8fdnd"] Oct 30 00:31:59.492637 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.492564 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f649b0-75dd767db7-8fdnd"] Oct 30 00:31:59.494353 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.494321 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvcrr\" (UniqueName: \"kubernetes.io/projected/4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c-kube-api-access-kvcrr\") pod \"maas-api-revoke-keys-e2e-praxis-b-d3e13d-gptcf\" (UID: \"4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3e13d-gptcf" Oct 30 00:31:59.494456 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.494442 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-d3e13d-gptcf\" (UID: \"4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3e13d-gptcf" Oct 30 00:31:59.494691 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.494673 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-d3e13d-gptcf\" (UID: \"4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3e13d-gptcf" Oct 30 00:31:59.495267 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.495243 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-d3e13d-gptcf\" (UID: \"4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3e13d-gptcf" Oct 30 00:31:59.497025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.497005 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-d3e13d-gptcf\" (UID: \"4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3e13d-gptcf" Oct 30 00:31:59.540556 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.540520 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvcrr\" (UniqueName: \"kubernetes.io/projected/4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c-kube-api-access-kvcrr\") pod \"maas-api-revoke-keys-e2e-praxis-b-d3e13d-gptcf\" (UID: \"4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3e13d-gptcf" Oct 30 00:31:59.580670 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.580631 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f649b0-57947cf4fd-qdghs"] Oct 30 00:31:59.588628 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.588416 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f649b0-57947cf4fd-qdghs"] Oct 30 00:31:59.590568 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.590501 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3e13d-gptcf" Oct 30 00:31:59.595317 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.595289 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65938fbc-9537-4aa2-b0b0-a77ff94384ca-kube-api-access-h7h9d\" (UniqueName: \"kubernetes.io/projected/65938fbc-9537-4aa2-b0b0-a77ff94384ca-kube-api-access-h7h9d\") pod \"65938fbc-9537-4aa2-b0b0-a77ff94384ca\" (UID: \"65938fbc-9537-4aa2-b0b0-a77ff94384ca\") " Oct 30 00:31:59.595578 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.595558 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65938fbc-9537-4aa2-b0b0-a77ff94384ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65938fbc-9537-4aa2-b0b0-a77ff94384ca-plugins-config-volume\") pod \"65938fbc-9537-4aa2-b0b0-a77ff94384ca\" (UID: \"65938fbc-9537-4aa2-b0b0-a77ff94384ca\") " Oct 30 00:31:59.597216 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.596599 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65938fbc-9537-4aa2-b0b0-a77ff94384ca-plugins-config-volume" pod "65938fbc-9537-4aa2-b0b0-a77ff94384ca" (UID: "65938fbc-9537-4aa2-b0b0-a77ff94384ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:31:59.598057 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.598030 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65938fbc-9537-4aa2-b0b0-a77ff94384ca-kube-api-access-h7h9d" pod "65938fbc-9537-4aa2-b0b0-a77ff94384ca" (UID: "65938fbc-9537-4aa2-b0b0-a77ff94384ca"). InnerVolumeSpecName "kube-api-access-h7h9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:31:59.705074 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.705036 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7h9d\" (UniqueName: \"kubernetes.io/projected/65938fbc-9537-4aa2-b0b0-a77ff94384ca-kube-api-access-h7h9d\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:31:59.705336 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:31:59.705310 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65938fbc-9537-4aa2-b0b0-a77ff94384ca-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:00.049769 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:00.049728 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3e13d-gptcf"] Oct 30 00:32:00.261538 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:00.261171 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:32:00.261538 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:00.261239 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" podUID="bdd71cc9-c2f9-4924-ab67-4b5f03b08f90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:32:00.464122 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:00.464076 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3e13d-gptcf" event={"ID":"4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c","Type":"ContainerStarted","Data":"f19c2b07581b55571cb674bde58e5104cd609e89c840fbb4b1488b6d6c2b212c"} Oct 30 00:32:00.470759 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:00.470723 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1dd8c1-755bcc85b6-2zdss" event={"ID":"65938fbc-9537-4aa2-b0b0-a77ff94384ca","Type":"ContainerDied","Data":"914c6948740952c3b5c1453aabcf57a56a28b5e1b300cbd3a791aa1f57e7f6e1"} Oct 30 00:32:00.470961 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:00.470786 5001 scope.go:122] "RemoveContainer" containerID="dddb0d1ab2dea99b9dc6eb5563c4f90eaa51c66c53f8f715b87feac27cc5d004" Oct 30 00:32:00.470961 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:00.470838 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1dd8c1-755bcc85b6-2zdss" Oct 30 00:32:00.538964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:00.538890 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1dd8c1-755bcc85b6-2zdss"] Oct 30 00:32:00.543548 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:00.543519 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1dd8c1-755bcc85b6-2zdss"] Oct 30 00:32:00.920491 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:00.920460 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4925fd35-f8b1-4706-990d-a291bb1ed760" path="/var/lib/kubelet/pods/4925fd35-f8b1-4706-990d-a291bb1ed760/volumes" Oct 30 00:32:00.920907 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:00.920891 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65938fbc-9537-4aa2-b0b0-a77ff94384ca" path="/var/lib/kubelet/pods/65938fbc-9537-4aa2-b0b0-a77ff94384ca/volumes" Oct 30 00:32:00.921330 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:00.921317 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="820af093-87be-4078-9098-1997a629c2c7" path="/var/lib/kubelet/pods/820af093-87be-4078-9098-1997a629c2c7/volumes" Oct 30 00:32:01.485900 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:01.485823 5001 generic.go:356] "Generic (PLEG): container finished" podID="4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c" containerID="d6f233579d5c6b19fcd5a7b398fc86f945e638ca70618baf7fed77261f565c15" exitCode=0 Oct 30 00:32:01.486078 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:01.485927 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3e13d-gptcf" event={"ID":"4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c","Type":"ContainerDied","Data":"d6f233579d5c6b19fcd5a7b398fc86f945e638ca70618baf7fed77261f565c15"} Oct 30 00:32:02.502743 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:02.502652 5001 generic.go:356] "Generic (PLEG): container finished" podID="bdd71cc9-c2f9-4924-ab67-4b5f03b08f90" containerID="be7a44843a754b3e018a34b63f1c16f6efb066a59e0d101031e550dd7a7808fe" exitCode=0 Oct 30 00:32:02.503323 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:02.503093 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" event={"ID":"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90","Type":"ContainerDied","Data":"be7a44843a754b3e018a34b63f1c16f6efb066a59e0d101031e550dd7a7808fe"} Oct 30 00:32:02.503802 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:02.503622 5001 scope.go:122] "RemoveContainer" containerID="be7a44843a754b3e018a34b63f1c16f6efb066a59e0d101031e550dd7a7808fe" Oct 30 00:32:02.506228 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:02.506184 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:32:02.506354 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:02.506289 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:32:02.506354 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:02.506339 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:32:03.003048 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:03.002964 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1dd8c1-x8z46"] Oct 30 00:32:03.127994 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:03.120749 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1dd8c1-x8z46"] Oct 30 00:32:03.268878 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:03.268545 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3e13d-gptcf" Oct 30 00:32:03.358278 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:03.355613 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c-maas-api-openshift-ca\") pod \"4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c\" (UID: \"4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c\") " Oct 30 00:32:03.358278 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:03.355810 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c-kube-api-access-kvcrr\" (UniqueName: \"kubernetes.io/projected/4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c-kube-api-access-kvcrr\") pod \"4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c\" (UID: \"4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c\") " Oct 30 00:32:03.358278 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:03.355885 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c-maas-api-xks-ca\") pod \"4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c\" (UID: \"4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c\") " Oct 30 00:32:03.362204 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:03.361765 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c-maas-api-openshift-ca" pod "4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c" (UID: "4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:32:03.411845 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:03.411489 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c-maas-api-xks-ca" pod "4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c" (UID: "4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:32:03.481664 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:03.480477 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:03.481664 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:03.480516 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:03.497291 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:03.494193 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c-kube-api-access-kvcrr" pod "4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c" (UID: "4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c"). InnerVolumeSpecName "kube-api-access-kvcrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:32:03.517077 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:03.517037 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" event={"ID":"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90","Type":"ContainerStarted","Data":"cf080347689293c5463f0ee735fe123c0cf17f251462ec940d395a664b0a2ec9"} Oct 30 00:32:03.519440 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:03.519418 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3e13d-gptcf" Oct 30 00:32:03.519560 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:03.519449 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3e13d-gptcf" event={"ID":"4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c","Type":"ContainerDied","Data":"f19c2b07581b55571cb674bde58e5104cd609e89c840fbb4b1488b6d6c2b212c"} Oct 30 00:32:03.519560 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:03.519501 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f19c2b07581b55571cb674bde58e5104cd609e89c840fbb4b1488b6d6c2b212c" Oct 30 00:32:03.582665 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:03.582624 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvcrr\" (UniqueName: \"kubernetes.io/projected/4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c-kube-api-access-kvcrr\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:03.801209 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:03.801174 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:32:03.801980 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:03.801950 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 00:32:03.802105 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:03.802013 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" podUID="bdd71cc9-c2f9-4924-ab67-4b5f03b08f90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 00:32:04.262831 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.262797 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac8930ef-x4xqc"] Oct 30 00:32:04.263476 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.263460 5001 cpu_manager.go:414] "containerMap: removing container" podUID="65938fbc-9537-4aa2-b0b0-a77ff94384ca" containerName="payload-processing" Oct 30 00:32:04.263542 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.263478 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65938fbc-9537-4aa2-b0b0-a77ff94384ca" containerName="payload-processing" Oct 30 00:32:04.263542 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.263522 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c" containerName="revoke-keys" Oct 30 00:32:04.263542 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.263527 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c" containerName="revoke-keys" Oct 30 00:32:04.263648 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.263629 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c" containerName="revoke-keys" Oct 30 00:32:04.263648 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.263641 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="65938fbc-9537-4aa2-b0b0-a77ff94384ca" containerName="payload-processing" Oct 30 00:32:04.267234 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.267216 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8930ef-x4xqc" Oct 30 00:32:04.282344 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.282323 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:32:04.292286 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.292265 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:32:04.292508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.292320 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:32:04.292586 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.292319 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:32:04.299681 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.299658 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac8930ef-x4xqc"] Oct 30 00:32:04.400333 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.400282 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac8930ef-x4xqc\" (UID: \"46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8930ef-x4xqc" Oct 30 00:32:04.400495 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.400440 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nmlf\" (UniqueName: \"kubernetes.io/projected/46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b-kube-api-access-2nmlf\") pod \"maas-api-revoke-sub-ac8930ef-x4xqc\" (UID: \"46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8930ef-x4xqc" Oct 30 00:32:04.400545 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.400525 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac8930ef-x4xqc\" (UID: \"46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8930ef-x4xqc" Oct 30 00:32:04.502524 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.502495 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac8930ef-x4xqc\" (UID: \"46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8930ef-x4xqc" Oct 30 00:32:04.502670 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.502617 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nmlf\" (UniqueName: \"kubernetes.io/projected/46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b-kube-api-access-2nmlf\") pod \"maas-api-revoke-sub-ac8930ef-x4xqc\" (UID: \"46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8930ef-x4xqc" Oct 30 00:32:04.502735 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.502690 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac8930ef-x4xqc\" (UID: \"46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8930ef-x4xqc" Oct 30 00:32:04.503488 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.503461 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac8930ef-x4xqc\" (UID: \"46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8930ef-x4xqc" Oct 30 00:32:04.505092 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.505066 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac8930ef-x4xqc\" (UID: \"46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8930ef-x4xqc" Oct 30 00:32:04.561945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.561863 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nmlf\" (UniqueName: \"kubernetes.io/projected/46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b-kube-api-access-2nmlf\") pod \"maas-api-revoke-sub-ac8930ef-x4xqc\" (UID: \"46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8930ef-x4xqc" Oct 30 00:32:04.595001 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.593498 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8930ef-x4xqc" Oct 30 00:32:04.697823 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.697762 5001 generic.go:356] "Generic (PLEG): container finished" podID="34da2ec4-9938-4eaf-9845-dc23bcbc761a" containerID="f10fe7ca539096970b05937017c10fc1e492f75a6174b548cc05fd9cfd2c712b" exitCode=0 Oct 30 00:32:04.698933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.698323 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" event={"ID":"34da2ec4-9938-4eaf-9845-dc23bcbc761a","Type":"ContainerDied","Data":"f10fe7ca539096970b05937017c10fc1e492f75a6174b548cc05fd9cfd2c712b"} Oct 30 00:32:04.698933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.698392 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" event={"ID":"34da2ec4-9938-4eaf-9845-dc23bcbc761a","Type":"ContainerDied","Data":"a78c6fc9271dfd313e8b4b436ce9cb990c9ceaa587409f5d4edc5f8a71e10f8e"} Oct 30 00:32:04.698933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.698419 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a78c6fc9271dfd313e8b4b436ce9cb990c9ceaa587409f5d4edc5f8a71e10f8e" Oct 30 00:32:04.698933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.698438 5001 scope.go:122] "RemoveContainer" containerID="74bd4f05adabe0d4f743035fa48fd38389be4628b8c007ae54ce96696cec5737" Oct 30 00:32:04.706784 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.706684 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:32:04.756452 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.755816 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istiod-ca-cert\") pod \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " Oct 30 00:32:04.756452 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.755891 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-workload-certs\") pod \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " Oct 30 00:32:04.756452 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.756061 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-credential-socket\") pod \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " Oct 30 00:32:04.756452 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.756158 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istio-podinfo\") pod \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " Oct 30 00:32:04.756452 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.756223 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-workload-socket\") pod \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " Oct 30 00:32:04.756452 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.756258 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istio-token\" (UniqueName: \"kubernetes.io/projected/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istio-token\") pod \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " Oct 30 00:32:04.756452 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.756333 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istio-data\") pod \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " Oct 30 00:32:04.756974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.756652 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-credential-socket" pod "34da2ec4-9938-4eaf-9845-dc23bcbc761a" (UID: "34da2ec4-9938-4eaf-9845-dc23bcbc761a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:04.756974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.756883 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-workload-socket" pod "34da2ec4-9938-4eaf-9845-dc23bcbc761a" (UID: "34da2ec4-9938-4eaf-9845-dc23bcbc761a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:04.756974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.756947 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-workload-certs" pod "34da2ec4-9938-4eaf-9845-dc23bcbc761a" (UID: "34da2ec4-9938-4eaf-9845-dc23bcbc761a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:04.757145 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.757059 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istiod-ca-cert" pod "34da2ec4-9938-4eaf-9845-dc23bcbc761a" (UID: "34da2ec4-9938-4eaf-9845-dc23bcbc761a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:32:04.757206 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.757169 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istio-data" pod "34da2ec4-9938-4eaf-9845-dc23bcbc761a" (UID: "34da2ec4-9938-4eaf-9845-dc23bcbc761a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:04.758811 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.757395 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istio-envoy\") pod \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " Oct 30 00:32:04.758811 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.757443 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34da2ec4-9938-4eaf-9845-dc23bcbc761a-kube-api-access-xbq45\" (UniqueName: \"kubernetes.io/projected/34da2ec4-9938-4eaf-9845-dc23bcbc761a-kube-api-access-xbq45\") pod \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\" (UID: \"34da2ec4-9938-4eaf-9845-dc23bcbc761a\") " Oct 30 00:32:04.758811 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.758160 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:04.758811 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.758181 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:04.758811 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.758195 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:04.758811 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.758207 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:04.758811 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.758219 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:04.758811 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.758783 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istio-podinfo" pod "34da2ec4-9938-4eaf-9845-dc23bcbc761a" (UID: "34da2ec4-9938-4eaf-9845-dc23bcbc761a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:32:04.760303 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.760248 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istio-envoy" pod "34da2ec4-9938-4eaf-9845-dc23bcbc761a" (UID: "34da2ec4-9938-4eaf-9845-dc23bcbc761a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:04.761692 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.761665 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istio-token" pod "34da2ec4-9938-4eaf-9845-dc23bcbc761a" (UID: "34da2ec4-9938-4eaf-9845-dc23bcbc761a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:32:04.763034 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.763009 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34da2ec4-9938-4eaf-9845-dc23bcbc761a-kube-api-access-xbq45" pod "34da2ec4-9938-4eaf-9845-dc23bcbc761a" (UID: "34da2ec4-9938-4eaf-9845-dc23bcbc761a"). InnerVolumeSpecName "kube-api-access-xbq45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:32:04.802946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.802250 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 00:32:04.802946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.802337 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" podUID="bdd71cc9-c2f9-4924-ab67-4b5f03b08f90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 00:32:04.859522 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.859489 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:04.859522 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.859517 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:04.859522 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.859529 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34da2ec4-9938-4eaf-9845-dc23bcbc761a-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:04.859824 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.859538 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbq45\" (UniqueName: \"kubernetes.io/projected/34da2ec4-9938-4eaf-9845-dc23bcbc761a-kube-api-access-xbq45\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:04.943306 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:04.943269 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd26fcf0-a65b-40e8-ad15-213b90e866bc" path="/var/lib/kubelet/pods/cd26fcf0-a65b-40e8-ad15-213b90e866bc/volumes" Oct 30 00:32:05.039312 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:32:05.039274 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46bbfb19_c05d_4ce5_b5ee_9f47e8618b9b.slice/crio-94269ea385cdee1915e9a16279b38b36b1a7f0603af8fd62a6fbb2bf3be74982 WatchSource:0}: Error finding container 94269ea385cdee1915e9a16279b38b36b1a7f0603af8fd62a6fbb2bf3be74982: Status 404 returned error can't find the container with id 94269ea385cdee1915e9a16279b38b36b1a7f0603af8fd62a6fbb2bf3be74982 Oct 30 00:32:05.041753 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:05.041719 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac8930ef-x4xqc"] Oct 30 00:32:05.773350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:05.767031 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn" Oct 30 00:32:05.780810 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:05.778666 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8930ef-x4xqc" event={"ID":"46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b","Type":"ContainerStarted","Data":"2cdaf15df360e278022cb90a80d05c2b8031f6f658894c01c5b65b66839c5732"} Oct 30 00:32:05.780810 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:05.778735 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8930ef-x4xqc" event={"ID":"46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b","Type":"ContainerStarted","Data":"94269ea385cdee1915e9a16279b38b36b1a7f0603af8fd62a6fbb2bf3be74982"} Oct 30 00:32:05.849975 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:05.841598 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 00:32:05.851959 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:05.850309 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" podUID="bdd71cc9-c2f9-4924-ab67-4b5f03b08f90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 00:32:05.873097 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:05.873059 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn"] Oct 30 00:32:05.877833 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:05.877807 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-93016b-openshift-default-6754fffd7c-lwnwn"] Oct 30 00:32:05.923934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:05.913124 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8930ef-x4xqc" podStartSLOduration=1.913104197 podStartE2EDuration="1.913104197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:05.907082862 +0000 UTC m=+1641.112584054" watchObservedRunningTime="2026-10-30 00:32:05.913104197 +0000 UTC m=+1641.118605376" Oct 30 00:32:06.789721 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:06.789689 5001 generic.go:356] "Generic (PLEG): container finished" podID="46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b" containerID="2cdaf15df360e278022cb90a80d05c2b8031f6f658894c01c5b65b66839c5732" exitCode=0 Oct 30 00:32:06.790118 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:06.789841 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8930ef-x4xqc" event={"ID":"46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b","Type":"ContainerDied","Data":"2cdaf15df360e278022cb90a80d05c2b8031f6f658894c01c5b65b66839c5732"} Oct 30 00:32:06.801513 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:06.801486 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 00:32:06.801639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:06.801543 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" podUID="bdd71cc9-c2f9-4924-ab67-4b5f03b08f90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 00:32:06.926610 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:06.926063 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34da2ec4-9938-4eaf-9845-dc23bcbc761a" path="/var/lib/kubelet/pods/34da2ec4-9938-4eaf-9845-dc23bcbc761a/volumes" Oct 30 00:32:07.207485 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.207411 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb"] Oct 30 00:32:07.216936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.208109 5001 cpu_manager.go:414] "containerMap: removing container" podUID="34da2ec4-9938-4eaf-9845-dc23bcbc761a" containerName="istio-proxy" Oct 30 00:32:07.217208 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.217185 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34da2ec4-9938-4eaf-9845-dc23bcbc761a" containerName="istio-proxy" Oct 30 00:32:07.217397 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.217383 5001 cpu_manager.go:414] "containerMap: removing container" podUID="34da2ec4-9938-4eaf-9845-dc23bcbc761a" containerName="istio-proxy" Oct 30 00:32:07.227811 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.227785 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34da2ec4-9938-4eaf-9845-dc23bcbc761a" containerName="istio-proxy" Oct 30 00:32:07.228241 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.228219 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="34da2ec4-9938-4eaf-9845-dc23bcbc761a" containerName="istio-proxy" Oct 30 00:32:07.228397 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.228383 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="34da2ec4-9938-4eaf-9845-dc23bcbc761a" containerName="istio-proxy" Oct 30 00:32:07.246443 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.246413 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb"] Oct 30 00:32:07.246634 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.246537 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:32:07.378997 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.378953 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:32:07.379168 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.379093 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:32:07.379267 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.379238 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:32:07.379334 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.379318 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1450166b-4504-4cb0-adac-c43c514a06fd-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:32:07.379403 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.379345 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:32:07.379461 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.379410 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1450166b-4504-4cb0-adac-c43c514a06fd-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:32:07.379461 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.379430 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1450166b-4504-4cb0-adac-c43c514a06fd-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:32:07.379610 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.379582 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:32:07.379726 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.379710 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8gt8\" (UniqueName: \"kubernetes.io/projected/1450166b-4504-4cb0-adac-c43c514a06fd-kube-api-access-x8gt8\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:32:07.481494 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.481411 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:32:07.481494 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.481450 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1450166b-4504-4cb0-adac-c43c514a06fd-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:32:07.481494 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.481467 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:32:07.481762 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.481517 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1450166b-4504-4cb0-adac-c43c514a06fd-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:32:07.481762 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.481544 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1450166b-4504-4cb0-adac-c43c514a06fd-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:32:07.481762 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.481654 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:32:07.481970 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.481768 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8gt8\" (UniqueName: \"kubernetes.io/projected/1450166b-4504-4cb0-adac-c43c514a06fd-kube-api-access-x8gt8\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:32:07.481970 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.481825 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:32:07.481970 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.481856 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-istio-data\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:32:07.481970 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.481957 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:32:07.482182 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.482049 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-workload-certs\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:32:07.482285 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.482268 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-credential-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:32:07.482328 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.482312 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-workload-socket\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:32:07.482373 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.482350 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1450166b-4504-4cb0-adac-c43c514a06fd-istiod-ca-cert\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:32:07.484013 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.483991 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-istio-envoy\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:32:07.484380 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.484363 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1450166b-4504-4cb0-adac-c43c514a06fd-istio-podinfo\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:32:07.502212 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.502181 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1450166b-4504-4cb0-adac-c43c514a06fd-istio-token\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:32:07.502423 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.502407 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8gt8\" (UniqueName: \"kubernetes.io/projected/1450166b-4504-4cb0-adac-c43c514a06fd-kube-api-access-x8gt8\") pod \"e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:32:07.569679 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.569379 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:32:07.801111 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.801078 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 00:32:07.801486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.801149 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" podUID="bdd71cc9-c2f9-4924-ab67-4b5f03b08f90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 00:32:07.803613 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.803589 5001 generic.go:356] "Generic (PLEG): container finished" podID="b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2" containerID="67bb7ed0f9f134c303e88e150a5b0bb7dbeb03b797af6df909b5320374e7ed95" exitCode=0 Oct 30 00:32:07.803763 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.803669 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" event={"ID":"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2","Type":"ContainerDied","Data":"67bb7ed0f9f134c303e88e150a5b0bb7dbeb03b797af6df909b5320374e7ed95"} Oct 30 00:32:07.804638 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.804171 5001 scope.go:122] "RemoveContainer" containerID="67bb7ed0f9f134c303e88e150a5b0bb7dbeb03b797af6df909b5320374e7ed95" Oct 30 00:32:07.806775 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.806739 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:32:07.806854 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.806834 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:32:07.806904 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:07.806889 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:32:08.057339 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:08.056676 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb"] Oct 30 00:32:08.058631 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:32:08.058602 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1450166b_4504_4cb0_adac_c43c514a06fd.slice/crio-630f313b7ad3ed4bc7206d9295f65500cad6ee6eebd0da03b196e07a94178c57 WatchSource:0}: Error finding container 630f313b7ad3ed4bc7206d9295f65500cad6ee6eebd0da03b196e07a94178c57: Status 404 returned error can't find the container with id 630f313b7ad3ed4bc7206d9295f65500cad6ee6eebd0da03b196e07a94178c57 Oct 30 00:32:08.063236 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:08.063164 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:32:08.063236 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:08.063231 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:32:08.063385 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:08.063259 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:32:08.184072 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:08.184042 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8930ef-x4xqc" Oct 30 00:32:08.301292 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:08.300388 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b-maas-api-xks-ca\") pod \"46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b\" (UID: \"46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b\") " Oct 30 00:32:08.301292 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:08.300453 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b-maas-api-openshift-ca\") pod \"46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b\" (UID: \"46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b\") " Oct 30 00:32:08.301292 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:08.300664 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b-kube-api-access-2nmlf\" (UniqueName: \"kubernetes.io/projected/46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b-kube-api-access-2nmlf\") pod \"46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b\" (UID: \"46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b\") " Oct 30 00:32:08.342406 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:08.342370 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b-kube-api-access-2nmlf" pod "46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b" (UID: "46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b"). InnerVolumeSpecName "kube-api-access-2nmlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:32:08.342639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:08.342497 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b-maas-api-openshift-ca" pod "46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b" (UID: "46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:32:08.349931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:08.347130 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b-maas-api-xks-ca" pod "46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b" (UID: "46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:32:08.403173 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:08.403117 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:08.403173 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:08.403144 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:08.403173 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:08.403156 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nmlf\" (UniqueName: \"kubernetes.io/projected/46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b-kube-api-access-2nmlf\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:08.804358 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:08.804282 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 00:32:08.804358 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:08.804333 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" podUID="bdd71cc9-c2f9-4924-ab67-4b5f03b08f90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 00:32:08.904986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:08.883134 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" event={"ID":"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2","Type":"ContainerStarted","Data":"f1941fcab3c6376a39215108f4fa4307581dee13b7bb854995203d78a4fc1337"} Oct 30 00:32:08.904986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:08.891144 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8930ef-x4xqc" event={"ID":"46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b","Type":"ContainerDied","Data":"94269ea385cdee1915e9a16279b38b36b1a7f0603af8fd62a6fbb2bf3be74982"} Oct 30 00:32:08.904986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:08.891210 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94269ea385cdee1915e9a16279b38b36b1a7f0603af8fd62a6fbb2bf3be74982" Oct 30 00:32:08.904986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:08.891346 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8930ef-x4xqc" Oct 30 00:32:08.904986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:08.895590 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" event={"ID":"1450166b-4504-4cb0-adac-c43c514a06fd","Type":"ContainerStarted","Data":"1541d19b136b61c458ea816c5d7b12672dd364fca4ad35a4aef374afd3a448c8"} Oct 30 00:32:08.904986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:08.895659 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" event={"ID":"1450166b-4504-4cb0-adac-c43c514a06fd","Type":"ContainerStarted","Data":"630f313b7ad3ed4bc7206d9295f65500cad6ee6eebd0da03b196e07a94178c57"} Oct 30 00:32:09.056400 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:09.052396 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" podStartSLOduration=2.05237551 podStartE2EDuration="2.05237551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:09.00374843 +0000 UTC m=+1644.209249611" watchObservedRunningTime="2026-10-30 00:32:09.05237551 +0000 UTC m=+1644.257876691" Oct 30 00:32:09.572026 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:09.571993 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:32:09.572960 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:09.572899 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 30 00:32:09.573038 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:09.572985 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" podUID="1450166b-4504-4cb0-adac-c43c514a06fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 30 00:32:09.604979 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:09.604946 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:32:09.606249 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:09.606224 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 30 00:32:09.606321 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:09.606280 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" podUID="b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 30 00:32:09.835992 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:09.820088 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:32:09.835992 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:09.828734 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:32:09.886240 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:09.886156 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:32:10.310904 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:10.310855 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg"] Oct 30 00:32:10.590078 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:10.588203 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 30 00:32:10.590078 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:10.588265 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" podUID="1450166b-4504-4cb0-adac-c43c514a06fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 30 00:32:10.612845 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:10.612275 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 30 00:32:10.612845 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:10.612330 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" podUID="b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 30 00:32:10.868990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:10.858972 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bc8b7d-8996p"] Oct 30 00:32:10.868990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:10.860058 5001 cpu_manager.go:414] "containerMap: removing container" podUID="46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b" containerName="revoke-keys" Oct 30 00:32:10.868990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:10.860083 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b" containerName="revoke-keys" Oct 30 00:32:10.868990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:10.860258 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b" containerName="revoke-keys" Oct 30 00:32:10.895566 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:10.895534 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bc8b7d-8996p" Oct 30 00:32:10.934957 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:10.909276 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:32:11.013565 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:10.951973 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:32:11.013565 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:10.958304 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:32:11.013565 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:10.958392 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:32:11.087641 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:11.087148 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:32:11.095449 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:11.094722 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4766ba0-9711-413b-a9ff-90b8c040b421-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bc8b7d-8996p\" (UID: \"b4766ba0-9711-413b-a9ff-90b8c040b421\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bc8b7d-8996p" Oct 30 00:32:11.095449 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:11.094868 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4766ba0-9711-413b-a9ff-90b8c040b421-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bc8b7d-8996p\" (UID: \"b4766ba0-9711-413b-a9ff-90b8c040b421\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bc8b7d-8996p" Oct 30 00:32:11.095449 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:11.095012 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv22v\" (UniqueName: \"kubernetes.io/projected/b4766ba0-9711-413b-a9ff-90b8c040b421-kube-api-access-zv22v\") pod \"maas-api-revoke-keys-e2e-praxis-b-bc8b7d-8996p\" (UID: \"b4766ba0-9711-413b-a9ff-90b8c040b421\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bc8b7d-8996p" Oct 30 00:32:11.176022 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:11.175906 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bc8b7d-8996p"] Oct 30 00:32:11.179443 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:11.179390 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" podUID="b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2" containerName="istio-proxy" containerID="cri-o://f1941fcab3c6376a39215108f4fa4307581dee13b7bb854995203d78a4fc1337" gracePeriod=30 Oct 30 00:32:11.196447 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:11.196414 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv22v\" (UniqueName: \"kubernetes.io/projected/b4766ba0-9711-413b-a9ff-90b8c040b421-kube-api-access-zv22v\") pod \"maas-api-revoke-keys-e2e-praxis-b-bc8b7d-8996p\" (UID: \"b4766ba0-9711-413b-a9ff-90b8c040b421\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bc8b7d-8996p" Oct 30 00:32:11.196571 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:11.196532 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4766ba0-9711-413b-a9ff-90b8c040b421-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bc8b7d-8996p\" (UID: \"b4766ba0-9711-413b-a9ff-90b8c040b421\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bc8b7d-8996p" Oct 30 00:32:11.196634 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:11.196620 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4766ba0-9711-413b-a9ff-90b8c040b421-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bc8b7d-8996p\" (UID: \"b4766ba0-9711-413b-a9ff-90b8c040b421\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bc8b7d-8996p" Oct 30 00:32:11.197540 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:11.197517 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4766ba0-9711-413b-a9ff-90b8c040b421-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bc8b7d-8996p\" (UID: \"b4766ba0-9711-413b-a9ff-90b8c040b421\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bc8b7d-8996p" Oct 30 00:32:11.199847 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:11.199251 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4766ba0-9711-413b-a9ff-90b8c040b421-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bc8b7d-8996p\" (UID: \"b4766ba0-9711-413b-a9ff-90b8c040b421\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bc8b7d-8996p" Oct 30 00:32:11.273006 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:11.272838 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv22v\" (UniqueName: \"kubernetes.io/projected/b4766ba0-9711-413b-a9ff-90b8c040b421-kube-api-access-zv22v\") pod \"maas-api-revoke-keys-e2e-praxis-b-bc8b7d-8996p\" (UID: \"b4766ba0-9711-413b-a9ff-90b8c040b421\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bc8b7d-8996p" Oct 30 00:32:11.478844 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:11.478750 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bc8b7d-8996p" Oct 30 00:32:11.572572 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:11.572526 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 30 00:32:11.572753 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:11.572609 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" podUID="1450166b-4504-4cb0-adac-c43c514a06fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 30 00:32:11.866995 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:11.866471 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bc8b7d-8996p"] Oct 30 00:32:12.378935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:12.378614 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91e3e499-kj5d7"] Oct 30 00:32:12.400192 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:12.400163 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e3e499-kj5d7" Oct 30 00:32:12.402360 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:12.402286 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bc8b7d-8996p" event={"ID":"b4766ba0-9711-413b-a9ff-90b8c040b421","Type":"ContainerStarted","Data":"df23a6bafaee9e357dd5f43adfaef87e40a55c82fed0c52149216b96d916ee4c"} Oct 30 00:32:12.405984 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:12.405881 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91e3e499-kj5d7"] Oct 30 00:32:12.570347 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:12.570312 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 30 00:32:12.570543 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:12.570364 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" podUID="1450166b-4504-4cb0-adac-c43c514a06fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 30 00:32:12.601772 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:12.601735 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30c22e7c-2be6-4b50-a71e-928adbe4196f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91e3e499-kj5d7\" (UID: \"30c22e7c-2be6-4b50-a71e-928adbe4196f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e3e499-kj5d7" Oct 30 00:32:12.601941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:12.601856 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30c22e7c-2be6-4b50-a71e-928adbe4196f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91e3e499-kj5d7\" (UID: \"30c22e7c-2be6-4b50-a71e-928adbe4196f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e3e499-kj5d7" Oct 30 00:32:12.602030 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:12.602010 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjbrw\" (UniqueName: \"kubernetes.io/projected/30c22e7c-2be6-4b50-a71e-928adbe4196f-kube-api-access-fjbrw\") pod \"maas-api-revoke-sub-91e3e499-kj5d7\" (UID: \"30c22e7c-2be6-4b50-a71e-928adbe4196f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e3e499-kj5d7" Oct 30 00:32:12.703891 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:12.703854 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30c22e7c-2be6-4b50-a71e-928adbe4196f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91e3e499-kj5d7\" (UID: \"30c22e7c-2be6-4b50-a71e-928adbe4196f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e3e499-kj5d7" Oct 30 00:32:12.704104 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:12.703978 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30c22e7c-2be6-4b50-a71e-928adbe4196f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91e3e499-kj5d7\" (UID: \"30c22e7c-2be6-4b50-a71e-928adbe4196f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e3e499-kj5d7" Oct 30 00:32:12.704104 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:12.704078 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjbrw\" (UniqueName: \"kubernetes.io/projected/30c22e7c-2be6-4b50-a71e-928adbe4196f-kube-api-access-fjbrw\") pod \"maas-api-revoke-sub-91e3e499-kj5d7\" (UID: \"30c22e7c-2be6-4b50-a71e-928adbe4196f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e3e499-kj5d7" Oct 30 00:32:12.704853 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:12.704822 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30c22e7c-2be6-4b50-a71e-928adbe4196f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91e3e499-kj5d7\" (UID: \"30c22e7c-2be6-4b50-a71e-928adbe4196f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e3e499-kj5d7" Oct 30 00:32:12.727124 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:12.727045 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:32:12.737934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:12.736679 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjbrw\" (UniqueName: \"kubernetes.io/projected/30c22e7c-2be6-4b50-a71e-928adbe4196f-kube-api-access-fjbrw\") pod \"maas-api-revoke-sub-91e3e499-kj5d7\" (UID: \"30c22e7c-2be6-4b50-a71e-928adbe4196f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e3e499-kj5d7" Oct 30 00:32:12.760934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:12.752321 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30c22e7c-2be6-4b50-a71e-928adbe4196f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91e3e499-kj5d7\" (UID: \"30c22e7c-2be6-4b50-a71e-928adbe4196f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e3e499-kj5d7" Oct 30 00:32:12.781213 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:12.781179 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e3e499-kj5d7" Oct 30 00:32:13.167040 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:13.167014 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91e3e499-kj5d7"] Oct 30 00:32:13.451938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:13.439727 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e3e499-kj5d7" event={"ID":"30c22e7c-2be6-4b50-a71e-928adbe4196f","Type":"ContainerStarted","Data":"cf81011bd2173405513ec2e615c7bea2580ef24fc53e1c45d88220ad7b7feced"} Oct 30 00:32:13.468304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:13.454862 5001 generic.go:356] "Generic (PLEG): container finished" podID="b4766ba0-9711-413b-a9ff-90b8c040b421" containerID="efc0d87efc297068e8a20f038be19301da0e59498f27199413b2b67ec19be094" exitCode=0 Oct 30 00:32:13.468304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:13.454984 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bc8b7d-8996p" event={"ID":"b4766ba0-9711-413b-a9ff-90b8c040b421","Type":"ContainerDied","Data":"efc0d87efc297068e8a20f038be19301da0e59498f27199413b2b67ec19be094"} Oct 30 00:32:13.597936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:13.597053 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 30 00:32:13.597936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:13.597127 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" podUID="1450166b-4504-4cb0-adac-c43c514a06fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 30 00:32:14.470535 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:14.470504 5001 generic.go:356] "Generic (PLEG): container finished" podID="30c22e7c-2be6-4b50-a71e-928adbe4196f" containerID="bcbc2a74bd72da2d3dc5c806eccd3d4e96061472f7bc0a0418c5c7792457dfb6" exitCode=0 Oct 30 00:32:14.471070 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:14.470594 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e3e499-kj5d7" event={"ID":"30c22e7c-2be6-4b50-a71e-928adbe4196f","Type":"ContainerDied","Data":"bcbc2a74bd72da2d3dc5c806eccd3d4e96061472f7bc0a0418c5c7792457dfb6"} Oct 30 00:32:14.570626 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:14.570595 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 30 00:32:14.570864 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:14.570841 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" podUID="1450166b-4504-4cb0-adac-c43c514a06fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 30 00:32:14.781008 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:14.780982 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bc8b7d-8996p" Oct 30 00:32:14.997614 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:14.994343 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4766ba0-9711-413b-a9ff-90b8c040b421-kube-api-access-zv22v\" (UniqueName: \"kubernetes.io/projected/b4766ba0-9711-413b-a9ff-90b8c040b421-kube-api-access-zv22v\") pod \"b4766ba0-9711-413b-a9ff-90b8c040b421\" (UID: \"b4766ba0-9711-413b-a9ff-90b8c040b421\") " Oct 30 00:32:14.997614 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:14.994504 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4766ba0-9711-413b-a9ff-90b8c040b421-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4766ba0-9711-413b-a9ff-90b8c040b421-maas-api-xks-ca\") pod \"b4766ba0-9711-413b-a9ff-90b8c040b421\" (UID: \"b4766ba0-9711-413b-a9ff-90b8c040b421\") " Oct 30 00:32:14.997614 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:14.994596 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4766ba0-9711-413b-a9ff-90b8c040b421-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4766ba0-9711-413b-a9ff-90b8c040b421-maas-api-openshift-ca\") pod \"b4766ba0-9711-413b-a9ff-90b8c040b421\" (UID: \"b4766ba0-9711-413b-a9ff-90b8c040b421\") " Oct 30 00:32:14.997614 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:14.996286 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4766ba0-9711-413b-a9ff-90b8c040b421-maas-api-openshift-ca" pod "b4766ba0-9711-413b-a9ff-90b8c040b421" (UID: "b4766ba0-9711-413b-a9ff-90b8c040b421"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:32:14.997614 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:14.996644 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4766ba0-9711-413b-a9ff-90b8c040b421-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:15.003931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:14.998412 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4766ba0-9711-413b-a9ff-90b8c040b421-maas-api-xks-ca" pod "b4766ba0-9711-413b-a9ff-90b8c040b421" (UID: "b4766ba0-9711-413b-a9ff-90b8c040b421"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:32:15.019458 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:15.019427 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4766ba0-9711-413b-a9ff-90b8c040b421-kube-api-access-zv22v" pod "b4766ba0-9711-413b-a9ff-90b8c040b421" (UID: "b4766ba0-9711-413b-a9ff-90b8c040b421"). InnerVolumeSpecName "kube-api-access-zv22v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:32:15.101494 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:15.101457 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv22v\" (UniqueName: \"kubernetes.io/projected/b4766ba0-9711-413b-a9ff-90b8c040b421-kube-api-access-zv22v\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:15.101494 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:15.101486 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4766ba0-9711-413b-a9ff-90b8c040b421-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:15.480449 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:15.480416 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bc8b7d-8996p" Oct 30 00:32:15.480924 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:15.480423 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bc8b7d-8996p" event={"ID":"b4766ba0-9711-413b-a9ff-90b8c040b421","Type":"ContainerDied","Data":"df23a6bafaee9e357dd5f43adfaef87e40a55c82fed0c52149216b96d916ee4c"} Oct 30 00:32:15.480924 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:15.480576 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df23a6bafaee9e357dd5f43adfaef87e40a55c82fed0c52149216b96d916ee4c" Oct 30 00:32:15.597130 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:15.582462 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 30 00:32:15.597130 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:15.582523 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" podUID="1450166b-4504-4cb0-adac-c43c514a06fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 30 00:32:15.691075 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:15.691047 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e3e499-kj5d7" Oct 30 00:32:15.712025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:15.711593 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30c22e7c-2be6-4b50-a71e-928adbe4196f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30c22e7c-2be6-4b50-a71e-928adbe4196f-maas-api-xks-ca\") pod \"30c22e7c-2be6-4b50-a71e-928adbe4196f\" (UID: \"30c22e7c-2be6-4b50-a71e-928adbe4196f\") " Oct 30 00:32:15.712576 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:15.712554 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30c22e7c-2be6-4b50-a71e-928adbe4196f-kube-api-access-fjbrw\" (UniqueName: \"kubernetes.io/projected/30c22e7c-2be6-4b50-a71e-928adbe4196f-kube-api-access-fjbrw\") pod \"30c22e7c-2be6-4b50-a71e-928adbe4196f\" (UID: \"30c22e7c-2be6-4b50-a71e-928adbe4196f\") " Oct 30 00:32:15.712803 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:15.712788 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30c22e7c-2be6-4b50-a71e-928adbe4196f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30c22e7c-2be6-4b50-a71e-928adbe4196f-maas-api-openshift-ca\") pod \"30c22e7c-2be6-4b50-a71e-928adbe4196f\" (UID: \"30c22e7c-2be6-4b50-a71e-928adbe4196f\") " Oct 30 00:32:15.713286 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:15.713243 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30c22e7c-2be6-4b50-a71e-928adbe4196f-maas-api-openshift-ca" pod "30c22e7c-2be6-4b50-a71e-928adbe4196f" (UID: "30c22e7c-2be6-4b50-a71e-928adbe4196f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:32:15.733276 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:15.733240 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30c22e7c-2be6-4b50-a71e-928adbe4196f-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:15.734340 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:15.734313 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c22e7c-2be6-4b50-a71e-928adbe4196f-kube-api-access-fjbrw" pod "30c22e7c-2be6-4b50-a71e-928adbe4196f" (UID: "30c22e7c-2be6-4b50-a71e-928adbe4196f"). InnerVolumeSpecName "kube-api-access-fjbrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:32:15.736509 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:15.736471 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30c22e7c-2be6-4b50-a71e-928adbe4196f-maas-api-xks-ca" pod "30c22e7c-2be6-4b50-a71e-928adbe4196f" (UID: "30c22e7c-2be6-4b50-a71e-928adbe4196f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:32:15.835070 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:15.835031 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30c22e7c-2be6-4b50-a71e-928adbe4196f-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:15.835070 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:15.835068 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjbrw\" (UniqueName: \"kubernetes.io/projected/30c22e7c-2be6-4b50-a71e-928adbe4196f-kube-api-access-fjbrw\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:16.666935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:16.665822 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3e13d-gptcf"] Oct 30 00:32:16.695931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:16.672298 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e3e499-kj5d7" event={"ID":"30c22e7c-2be6-4b50-a71e-928adbe4196f","Type":"ContainerDied","Data":"cf81011bd2173405513ec2e615c7bea2580ef24fc53e1c45d88220ad7b7feced"} Oct 30 00:32:16.695931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:16.672360 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf81011bd2173405513ec2e615c7bea2580ef24fc53e1c45d88220ad7b7feced" Oct 30 00:32:16.695931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:16.672434 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e3e499-kj5d7" Oct 30 00:32:16.751443 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:16.751280 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d3e13d-gptcf"] Oct 30 00:32:16.970985 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:16.968879 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c" path="/var/lib/kubelet/pods/4bcd8bc0-4c34-45e4-8667-9294ac3c5a9c/volumes" Oct 30 00:32:17.081118 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:17.081075 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:32:17.081624 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:17.081596 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:32:17.082432 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:17.082413 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:32:17.324359 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:32:17.324325 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:32:20.370447 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.370384 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:32:20.502731 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.502696 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-kube-api-access-rxwc4\" (UniqueName: \"kubernetes.io/projected/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-kube-api-access-rxwc4\") pod \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " Oct 30 00:32:20.502898 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.502794 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istio-data\") pod \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " Oct 30 00:32:20.503000 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.502901 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istio-envoy\") pod \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " Oct 30 00:32:20.503000 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.502989 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-credential-socket\") pod \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " Oct 30 00:32:20.503095 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.503044 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istio-podinfo\") pod \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " Oct 30 00:32:20.503136 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.503106 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istio-data" pod "b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2" (UID: "b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:20.503194 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.503179 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istiod-ca-cert\") pod \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " Oct 30 00:32:20.503259 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.503213 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-workload-certs\") pod \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " Oct 30 00:32:20.503259 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.503244 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-workload-socket\") pod \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " Oct 30 00:32:20.503259 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.503246 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-credential-socket" pod "b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2" (UID: "b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:20.503788 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.503457 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istio-token\" (UniqueName: \"kubernetes.io/projected/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istio-token\") pod \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\" (UID: \"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2\") " Oct 30 00:32:20.503788 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.503485 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-workload-socket" pod "b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2" (UID: "b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:20.503788 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.503496 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-workload-certs" pod "b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2" (UID: "b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:20.503788 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.503585 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istiod-ca-cert" pod "b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2" (UID: "b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:32:20.504199 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.504177 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:20.504261 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.504207 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:20.504261 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.504221 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:20.504261 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.504235 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:20.504261 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.504251 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:20.505220 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.505194 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istio-podinfo" pod "b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2" (UID: "b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:32:20.505332 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.505246 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-kube-api-access-rxwc4" pod "b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2" (UID: "b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2"). InnerVolumeSpecName "kube-api-access-rxwc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:32:20.505417 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.505399 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istio-envoy" pod "b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2" (UID: "b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:20.505483 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.505471 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istio-token" pod "b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2" (UID: "b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:32:20.605787 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.605759 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:20.605787 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.605787 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxwc4\" (UniqueName: \"kubernetes.io/projected/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-kube-api-access-rxwc4\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:20.605981 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.605799 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:20.605981 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.605812 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:20.708691 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.708612 5001 generic.go:356] "Generic (PLEG): container finished" podID="b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2" containerID="f1941fcab3c6376a39215108f4fa4307581dee13b7bb854995203d78a4fc1337" exitCode=0 Oct 30 00:32:20.708691 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.708661 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" Oct 30 00:32:20.708876 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.708699 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" event={"ID":"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2","Type":"ContainerDied","Data":"f1941fcab3c6376a39215108f4fa4307581dee13b7bb854995203d78a4fc1337"} Oct 30 00:32:20.708876 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.708754 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg" event={"ID":"b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2","Type":"ContainerDied","Data":"2ffcd9668c840d886c6372c1a2003e72e36985ec7bc94e28f044ec606957d0ff"} Oct 30 00:32:20.708876 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.708775 5001 scope.go:122] "RemoveContainer" containerID="f1941fcab3c6376a39215108f4fa4307581dee13b7bb854995203d78a4fc1337" Oct 30 00:32:20.721440 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.721421 5001 scope.go:122] "RemoveContainer" containerID="67bb7ed0f9f134c303e88e150a5b0bb7dbeb03b797af6df909b5320374e7ed95" Oct 30 00:32:20.731428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.731405 5001 scope.go:122] "RemoveContainer" containerID="f1941fcab3c6376a39215108f4fa4307581dee13b7bb854995203d78a4fc1337" Oct 30 00:32:20.731711 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:32:20.731683 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1941fcab3c6376a39215108f4fa4307581dee13b7bb854995203d78a4fc1337\": container with ID starting with f1941fcab3c6376a39215108f4fa4307581dee13b7bb854995203d78a4fc1337 not found: ID does not exist" containerID="f1941fcab3c6376a39215108f4fa4307581dee13b7bb854995203d78a4fc1337" Oct 30 00:32:20.731770 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.731709 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1941fcab3c6376a39215108f4fa4307581dee13b7bb854995203d78a4fc1337"} err="failed to get container status \"f1941fcab3c6376a39215108f4fa4307581dee13b7bb854995203d78a4fc1337\": rpc error: code = NotFound desc = could not find container \"f1941fcab3c6376a39215108f4fa4307581dee13b7bb854995203d78a4fc1337\": container with ID starting with f1941fcab3c6376a39215108f4fa4307581dee13b7bb854995203d78a4fc1337 not found: ID does not exist" Oct 30 00:32:20.731770 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.731727 5001 scope.go:122] "RemoveContainer" containerID="67bb7ed0f9f134c303e88e150a5b0bb7dbeb03b797af6df909b5320374e7ed95" Oct 30 00:32:20.732059 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:32:20.732032 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67bb7ed0f9f134c303e88e150a5b0bb7dbeb03b797af6df909b5320374e7ed95\": container with ID starting with 67bb7ed0f9f134c303e88e150a5b0bb7dbeb03b797af6df909b5320374e7ed95 not found: ID does not exist" containerID="67bb7ed0f9f134c303e88e150a5b0bb7dbeb03b797af6df909b5320374e7ed95" Oct 30 00:32:20.732164 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.732066 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67bb7ed0f9f134c303e88e150a5b0bb7dbeb03b797af6df909b5320374e7ed95"} err="failed to get container status \"67bb7ed0f9f134c303e88e150a5b0bb7dbeb03b797af6df909b5320374e7ed95\": rpc error: code = NotFound desc = could not find container \"67bb7ed0f9f134c303e88e150a5b0bb7dbeb03b797af6df909b5320374e7ed95\": container with ID starting with 67bb7ed0f9f134c303e88e150a5b0bb7dbeb03b797af6df909b5320374e7ed95 not found: ID does not exist" Oct 30 00:32:20.769543 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.769516 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg"] Oct 30 00:32:20.778438 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.778414 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-1dd8c1-openshift-default-66cff96f49-4f7gg"] Oct 30 00:32:20.919287 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:20.919253 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2" path="/var/lib/kubelet/pods/b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2/volumes" Oct 30 00:32:24.031803 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:24.031755 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2"] Oct 30 00:32:24.079964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:24.079711 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" podUID="bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6" containerName="istio-proxy" containerID="cri-o://6e24e1e71d99bfc753ed6edeee268e066803b81ec9f2b578341e7f9966b98114" gracePeriod=30 Oct 30 00:32:24.684999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:24.684962 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdf3c170-stq7z"] Oct 30 00:32:24.686235 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:24.686205 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b4766ba0-9711-413b-a9ff-90b8c040b421" containerName="revoke-keys" Oct 30 00:32:24.703991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:24.703954 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4766ba0-9711-413b-a9ff-90b8c040b421" containerName="revoke-keys" Oct 30 00:32:24.704253 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:24.704233 5001 cpu_manager.go:414] "containerMap: removing container" podUID="30c22e7c-2be6-4b50-a71e-928adbe4196f" containerName="revoke-keys" Oct 30 00:32:24.704405 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:24.704384 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30c22e7c-2be6-4b50-a71e-928adbe4196f" containerName="revoke-keys" Oct 30 00:32:24.704564 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:24.704553 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2" containerName="istio-proxy" Oct 30 00:32:24.704638 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:24.704628 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2" containerName="istio-proxy" Oct 30 00:32:24.704731 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:24.704722 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2" containerName="istio-proxy" Oct 30 00:32:24.704803 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:24.704793 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2" containerName="istio-proxy" Oct 30 00:32:24.705169 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:24.705146 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2" containerName="istio-proxy" Oct 30 00:32:24.705327 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:24.705308 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9aad1e3-a17c-405b-9f00-3d4da7d3ffe2" containerName="istio-proxy" Oct 30 00:32:24.705450 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:24.705432 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4766ba0-9711-413b-a9ff-90b8c040b421" containerName="revoke-keys" Oct 30 00:32:24.705551 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:24.705540 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="30c22e7c-2be6-4b50-a71e-928adbe4196f" containerName="revoke-keys" Oct 30 00:32:24.772157 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:24.772122 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdf3c170-stq7z"] Oct 30 00:32:24.772479 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:24.772457 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdf3c170-stq7z" Oct 30 00:32:24.795951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:24.793681 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:32:24.795951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:24.794449 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:32:24.795951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:24.794530 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:32:24.911936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:24.907844 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2f023ad-b552-4c5a-89bf-aac6755064f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdf3c170-stq7z\" (UID: \"f2f023ad-b552-4c5a-89bf-aac6755064f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdf3c170-stq7z" Oct 30 00:32:24.956055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:24.954215 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:32:25.025288 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:25.018640 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:32:25.072679 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:25.072634 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw9xd\" (UniqueName: \"kubernetes.io/projected/f2f023ad-b552-4c5a-89bf-aac6755064f1-kube-api-access-pw9xd\") pod \"maas-api-revoke-sub-bdf3c170-stq7z\" (UID: \"f2f023ad-b552-4c5a-89bf-aac6755064f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdf3c170-stq7z" Oct 30 00:32:25.073198 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:25.072775 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2f023ad-b552-4c5a-89bf-aac6755064f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdf3c170-stq7z\" (UID: \"f2f023ad-b552-4c5a-89bf-aac6755064f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdf3c170-stq7z" Oct 30 00:32:25.183457 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:25.183414 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2f023ad-b552-4c5a-89bf-aac6755064f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdf3c170-stq7z\" (UID: \"f2f023ad-b552-4c5a-89bf-aac6755064f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdf3c170-stq7z" Oct 30 00:32:25.183864 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:25.183836 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw9xd\" (UniqueName: \"kubernetes.io/projected/f2f023ad-b552-4c5a-89bf-aac6755064f1-kube-api-access-pw9xd\") pod \"maas-api-revoke-sub-bdf3c170-stq7z\" (UID: \"f2f023ad-b552-4c5a-89bf-aac6755064f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdf3c170-stq7z" Oct 30 00:32:25.184124 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:25.184104 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2f023ad-b552-4c5a-89bf-aac6755064f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdf3c170-stq7z\" (UID: \"f2f023ad-b552-4c5a-89bf-aac6755064f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdf3c170-stq7z" Oct 30 00:32:25.191865 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:25.185153 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2f023ad-b552-4c5a-89bf-aac6755064f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdf3c170-stq7z\" (UID: \"f2f023ad-b552-4c5a-89bf-aac6755064f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdf3c170-stq7z" Oct 30 00:32:25.228025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:25.209008 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bc8b7d-8996p"] Oct 30 00:32:25.248865 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:25.228340 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:32:25.249409 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:25.249380 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw9xd\" (UniqueName: \"kubernetes.io/projected/f2f023ad-b552-4c5a-89bf-aac6755064f1-kube-api-access-pw9xd\") pod \"maas-api-revoke-sub-bdf3c170-stq7z\" (UID: \"f2f023ad-b552-4c5a-89bf-aac6755064f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdf3c170-stq7z" Oct 30 00:32:25.263104 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:25.260682 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:32:25.306347 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:25.304226 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:32:25.306347 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:25.304641 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2f023ad-b552-4c5a-89bf-aac6755064f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdf3c170-stq7z\" (UID: \"f2f023ad-b552-4c5a-89bf-aac6755064f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdf3c170-stq7z" Oct 30 00:32:25.349538 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:25.345803 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdf3c170-stq7z" Oct 30 00:32:25.397939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:25.372556 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bc8b7d-8996p"] Oct 30 00:32:25.445191 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:25.410018 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 00:32:25.566693 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:25.566657 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8e7fdd9-tlbpv"] Oct 30 00:32:25.612270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:25.598868 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e7fdd9-tlbpv" Oct 30 00:32:25.618536 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:25.618507 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8e7fdd9-tlbpv"] Oct 30 00:32:25.743803 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:25.743754 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf7g7\" (UniqueName: \"kubernetes.io/projected/1832bfed-6454-4368-bcd5-223c048fcc36-kube-api-access-mf7g7\") pod \"maas-api-revoke-sub-f8e7fdd9-tlbpv\" (UID: \"1832bfed-6454-4368-bcd5-223c048fcc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e7fdd9-tlbpv" Oct 30 00:32:25.743974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:25.743923 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1832bfed-6454-4368-bcd5-223c048fcc36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8e7fdd9-tlbpv\" (UID: \"1832bfed-6454-4368-bcd5-223c048fcc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e7fdd9-tlbpv" Oct 30 00:32:25.743974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:25.743968 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1832bfed-6454-4368-bcd5-223c048fcc36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8e7fdd9-tlbpv\" (UID: \"1832bfed-6454-4368-bcd5-223c048fcc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e7fdd9-tlbpv" Oct 30 00:32:25.804986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:25.804957 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdf3c170-stq7z"] Oct 30 00:32:25.850390 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:25.850360 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1832bfed-6454-4368-bcd5-223c048fcc36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8e7fdd9-tlbpv\" (UID: \"1832bfed-6454-4368-bcd5-223c048fcc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e7fdd9-tlbpv" Oct 30 00:32:25.850592 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:25.850571 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf7g7\" (UniqueName: \"kubernetes.io/projected/1832bfed-6454-4368-bcd5-223c048fcc36-kube-api-access-mf7g7\") pod \"maas-api-revoke-sub-f8e7fdd9-tlbpv\" (UID: \"1832bfed-6454-4368-bcd5-223c048fcc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e7fdd9-tlbpv" Oct 30 00:32:25.850730 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:25.850674 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1832bfed-6454-4368-bcd5-223c048fcc36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8e7fdd9-tlbpv\" (UID: \"1832bfed-6454-4368-bcd5-223c048fcc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e7fdd9-tlbpv" Oct 30 00:32:25.851197 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:25.851177 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1832bfed-6454-4368-bcd5-223c048fcc36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8e7fdd9-tlbpv\" (UID: \"1832bfed-6454-4368-bcd5-223c048fcc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e7fdd9-tlbpv" Oct 30 00:32:25.853379 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:25.853353 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1832bfed-6454-4368-bcd5-223c048fcc36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8e7fdd9-tlbpv\" (UID: \"1832bfed-6454-4368-bcd5-223c048fcc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e7fdd9-tlbpv" Oct 30 00:32:25.871311 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:25.871287 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf7g7\" (UniqueName: \"kubernetes.io/projected/1832bfed-6454-4368-bcd5-223c048fcc36-kube-api-access-mf7g7\") pod \"maas-api-revoke-sub-f8e7fdd9-tlbpv\" (UID: \"1832bfed-6454-4368-bcd5-223c048fcc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e7fdd9-tlbpv" Oct 30 00:32:25.920133 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:25.920101 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e7fdd9-tlbpv" Oct 30 00:32:26.225860 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:32:26.225821 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1832bfed_6454_4368_bcd5_223c048fcc36.slice/crio-db673f5acb3bd72c5950455806c72f576527104e074225beb4f2e28e97e0681c WatchSource:0}: Error finding container db673f5acb3bd72c5950455806c72f576527104e074225beb4f2e28e97e0681c: Status 404 returned error can't find the container with id db673f5acb3bd72c5950455806c72f576527104e074225beb4f2e28e97e0681c Oct 30 00:32:26.227336 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:26.227309 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8e7fdd9-tlbpv"] Oct 30 00:32:26.360100 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:26.360064 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e7fdd9-tlbpv" event={"ID":"1832bfed-6454-4368-bcd5-223c048fcc36","Type":"ContainerStarted","Data":"db673f5acb3bd72c5950455806c72f576527104e074225beb4f2e28e97e0681c"} Oct 30 00:32:26.361825 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:26.361793 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdf3c170-stq7z" event={"ID":"f2f023ad-b552-4c5a-89bf-aac6755064f1","Type":"ContainerStarted","Data":"647a01a83022f6a61d2e8410720a2864332b4536d0ca6ba210b67eef678e81b7"} Oct 30 00:32:26.361975 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:26.361841 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdf3c170-stq7z" event={"ID":"f2f023ad-b552-4c5a-89bf-aac6755064f1","Type":"ContainerStarted","Data":"beac8c54f6ad572d438367748f2aa6b16345050974a580b785f4a28a59078152"} Oct 30 00:32:26.403066 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:26.402973 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdf3c170-stq7z" podStartSLOduration=2.402960453 podStartE2EDuration="2.402960453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:26.400685769 +0000 UTC m=+1661.606186950" watchObservedRunningTime="2026-10-30 00:32:26.402960453 +0000 UTC m=+1661.608461633" Oct 30 00:32:26.920842 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:26.920810 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4766ba0-9711-413b-a9ff-90b8c040b421" path="/var/lib/kubelet/pods/b4766ba0-9711-413b-a9ff-90b8c040b421/volumes" Oct 30 00:32:27.371990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:27.371956 5001 generic.go:356] "Generic (PLEG): container finished" podID="f2f023ad-b552-4c5a-89bf-aac6755064f1" containerID="647a01a83022f6a61d2e8410720a2864332b4536d0ca6ba210b67eef678e81b7" exitCode=0 Oct 30 00:32:27.372455 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:27.372079 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdf3c170-stq7z" event={"ID":"f2f023ad-b552-4c5a-89bf-aac6755064f1","Type":"ContainerDied","Data":"647a01a83022f6a61d2e8410720a2864332b4536d0ca6ba210b67eef678e81b7"} Oct 30 00:32:27.373942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:27.373921 5001 generic.go:356] "Generic (PLEG): container finished" podID="1832bfed-6454-4368-bcd5-223c048fcc36" containerID="54fc1e319135f739cd00341b547b519f8ba61ba86ce7a45dc3d9f5c9fe2fef73" exitCode=0 Oct 30 00:32:27.374072 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:27.373989 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e7fdd9-tlbpv" event={"ID":"1832bfed-6454-4368-bcd5-223c048fcc36","Type":"ContainerDied","Data":"54fc1e319135f739cd00341b547b519f8ba61ba86ce7a45dc3d9f5c9fe2fef73"} Oct 30 00:32:28.567888 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:28.567861 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e7fdd9-tlbpv" Oct 30 00:32:28.599135 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:28.599110 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdf3c170-stq7z" Oct 30 00:32:28.608681 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:28.608655 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1832bfed-6454-4368-bcd5-223c048fcc36-kube-api-access-mf7g7\" (UniqueName: \"kubernetes.io/projected/1832bfed-6454-4368-bcd5-223c048fcc36-kube-api-access-mf7g7\") pod \"1832bfed-6454-4368-bcd5-223c048fcc36\" (UID: \"1832bfed-6454-4368-bcd5-223c048fcc36\") " Oct 30 00:32:28.608822 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:28.608771 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2f023ad-b552-4c5a-89bf-aac6755064f1-kube-api-access-pw9xd\" (UniqueName: \"kubernetes.io/projected/f2f023ad-b552-4c5a-89bf-aac6755064f1-kube-api-access-pw9xd\") pod \"f2f023ad-b552-4c5a-89bf-aac6755064f1\" (UID: \"f2f023ad-b552-4c5a-89bf-aac6755064f1\") " Oct 30 00:32:28.608822 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:28.608806 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2f023ad-b552-4c5a-89bf-aac6755064f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2f023ad-b552-4c5a-89bf-aac6755064f1-maas-api-openshift-ca\") pod \"f2f023ad-b552-4c5a-89bf-aac6755064f1\" (UID: \"f2f023ad-b552-4c5a-89bf-aac6755064f1\") " Oct 30 00:32:28.608953 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:28.608888 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1832bfed-6454-4368-bcd5-223c048fcc36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1832bfed-6454-4368-bcd5-223c048fcc36-maas-api-xks-ca\") pod \"1832bfed-6454-4368-bcd5-223c048fcc36\" (UID: \"1832bfed-6454-4368-bcd5-223c048fcc36\") " Oct 30 00:32:28.609021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:28.609001 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2f023ad-b552-4c5a-89bf-aac6755064f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2f023ad-b552-4c5a-89bf-aac6755064f1-maas-api-xks-ca\") pod \"f2f023ad-b552-4c5a-89bf-aac6755064f1\" (UID: \"f2f023ad-b552-4c5a-89bf-aac6755064f1\") " Oct 30 00:32:28.609450 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:28.609122 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1832bfed-6454-4368-bcd5-223c048fcc36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1832bfed-6454-4368-bcd5-223c048fcc36-maas-api-openshift-ca\") pod \"1832bfed-6454-4368-bcd5-223c048fcc36\" (UID: \"1832bfed-6454-4368-bcd5-223c048fcc36\") " Oct 30 00:32:28.609450 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:28.609414 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2f023ad-b552-4c5a-89bf-aac6755064f1-maas-api-openshift-ca" pod "f2f023ad-b552-4c5a-89bf-aac6755064f1" (UID: "f2f023ad-b552-4c5a-89bf-aac6755064f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:32:28.609713 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:28.609689 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1832bfed-6454-4368-bcd5-223c048fcc36-maas-api-openshift-ca" pod "1832bfed-6454-4368-bcd5-223c048fcc36" (UID: "1832bfed-6454-4368-bcd5-223c048fcc36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:32:28.610005 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:28.609984 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2f023ad-b552-4c5a-89bf-aac6755064f1-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:28.610122 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:28.610007 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1832bfed-6454-4368-bcd5-223c048fcc36-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:28.611447 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:28.611405 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2f023ad-b552-4c5a-89bf-aac6755064f1-kube-api-access-pw9xd" pod "f2f023ad-b552-4c5a-89bf-aac6755064f1" (UID: "f2f023ad-b552-4c5a-89bf-aac6755064f1"). InnerVolumeSpecName "kube-api-access-pw9xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:32:28.611653 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:28.611629 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1832bfed-6454-4368-bcd5-223c048fcc36-maas-api-xks-ca" pod "1832bfed-6454-4368-bcd5-223c048fcc36" (UID: "1832bfed-6454-4368-bcd5-223c048fcc36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:32:28.612282 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:28.612264 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2f023ad-b552-4c5a-89bf-aac6755064f1-maas-api-xks-ca" pod "f2f023ad-b552-4c5a-89bf-aac6755064f1" (UID: "f2f023ad-b552-4c5a-89bf-aac6755064f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:32:28.612416 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:28.612391 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1832bfed-6454-4368-bcd5-223c048fcc36-kube-api-access-mf7g7" pod "1832bfed-6454-4368-bcd5-223c048fcc36" (UID: "1832bfed-6454-4368-bcd5-223c048fcc36"). InnerVolumeSpecName "kube-api-access-mf7g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:32:28.711735 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:28.711643 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf7g7\" (UniqueName: \"kubernetes.io/projected/1832bfed-6454-4368-bcd5-223c048fcc36-kube-api-access-mf7g7\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:28.711735 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:28.711672 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw9xd\" (UniqueName: \"kubernetes.io/projected/f2f023ad-b552-4c5a-89bf-aac6755064f1-kube-api-access-pw9xd\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:28.711735 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:28.711682 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1832bfed-6454-4368-bcd5-223c048fcc36-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:28.711735 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:28.711691 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2f023ad-b552-4c5a-89bf-aac6755064f1-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:29.398581 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.397825 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e7fdd9-tlbpv" Oct 30 00:32:29.398581 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.397854 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e7fdd9-tlbpv" event={"ID":"1832bfed-6454-4368-bcd5-223c048fcc36","Type":"ContainerDied","Data":"db673f5acb3bd72c5950455806c72f576527104e074225beb4f2e28e97e0681c"} Oct 30 00:32:29.398581 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.397972 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db673f5acb3bd72c5950455806c72f576527104e074225beb4f2e28e97e0681c" Oct 30 00:32:29.400299 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.400274 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdf3c170-stq7z" event={"ID":"f2f023ad-b552-4c5a-89bf-aac6755064f1","Type":"ContainerDied","Data":"beac8c54f6ad572d438367748f2aa6b16345050974a580b785f4a28a59078152"} Oct 30 00:32:29.400452 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.400318 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="beac8c54f6ad572d438367748f2aa6b16345050974a580b785f4a28a59078152" Oct 30 00:32:29.400452 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.400382 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdf3c170-stq7z" Oct 30 00:32:29.408846 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.403713 5001 generic.go:356] "Generic (PLEG): container finished" podID="bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6" containerID="6e24e1e71d99bfc753ed6edeee268e066803b81ec9f2b578341e7f9966b98114" exitCode=0 Oct 30 00:32:29.408846 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.403971 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" event={"ID":"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6","Type":"ContainerDied","Data":"6e24e1e71d99bfc753ed6edeee268e066803b81ec9f2b578341e7f9966b98114"} Oct 30 00:32:29.408846 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.404022 5001 scope.go:122] "RemoveContainer" containerID="02b8bbb99fd50c202822d113a2aa0456014e60c70caaaa058a5d9e9a665e4e87" Oct 30 00:32:29.424877 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.424860 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:32:29.562941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.552302 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istio-token\" (UniqueName: \"kubernetes.io/projected/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istio-token\") pod \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " Oct 30 00:32:29.562941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.552386 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istio-envoy\") pod \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " Oct 30 00:32:29.562941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.552475 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istio-podinfo\") pod \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " Oct 30 00:32:29.562941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.552507 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istiod-ca-cert\") pod \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " Oct 30 00:32:29.562941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.552562 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-workload-certs\") pod \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " Oct 30 00:32:29.562941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.552604 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-workload-socket\") pod \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " Oct 30 00:32:29.562941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.552686 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-kube-api-access-7bqwc\" (UniqueName: \"kubernetes.io/projected/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-kube-api-access-7bqwc\") pod \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " Oct 30 00:32:29.562941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.552868 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-credential-socket\") pod \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " Oct 30 00:32:29.562941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.552929 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istio-data\") pod \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\" (UID: \"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6\") " Oct 30 00:32:29.562941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.557801 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istio-data" pod "bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6" (UID: "bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:29.562941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.558393 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-workload-certs" pod "bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6" (UID: "bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:29.562941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.558682 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-workload-socket" pod "bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6" (UID: "bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:29.562941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.562278 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-credential-socket" pod "bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6" (UID: "bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:29.570526 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.570494 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istiod-ca-cert" pod "bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6" (UID: "bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:32:29.582733 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.579308 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istio-podinfo" pod "bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6" (UID: "bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:32:29.597108 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.596949 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-kube-api-access-7bqwc" pod "bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6" (UID: "bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6"). InnerVolumeSpecName "kube-api-access-7bqwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:32:29.607974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.607894 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istio-envoy" pod "bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6" (UID: "bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:29.607974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.607931 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istio-token" pod "bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6" (UID: "bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:32:29.658218 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.658186 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:29.658218 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.658216 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:29.658218 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.658225 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:29.658502 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.658233 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:29.658502 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.658243 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:29.658502 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.658250 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:29.658502 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.658259 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bqwc\" (UniqueName: \"kubernetes.io/projected/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-kube-api-access-7bqwc\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:29.658502 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.658269 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:29.658502 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:29.658277 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:30.414177 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:30.414148 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" Oct 30 00:32:30.414389 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:30.414178 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2" event={"ID":"bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6","Type":"ContainerDied","Data":"c99a98fa006c35d537b5f052547f846f673e2ec42f05e76a52d2f5ee91d3c033"} Oct 30 00:32:30.414389 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:30.414242 5001 scope.go:122] "RemoveContainer" containerID="6e24e1e71d99bfc753ed6edeee268e066803b81ec9f2b578341e7f9966b98114" Oct 30 00:32:30.473606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:30.473567 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2"] Oct 30 00:32:30.483060 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:30.483035 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-d3e13d-openshift-default-6dd7d97c77-6wsg2"] Oct 30 00:32:30.947939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:30.947496 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6" path="/var/lib/kubelet/pods/bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6/volumes" Oct 30 00:32:32.069669 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:32.069405 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k"] Oct 30 00:32:32.070197 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:32.069677 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" podUID="bdd71cc9-c2f9-4924-ab67-4b5f03b08f90" containerName="istio-proxy" containerID="cri-o://cf080347689293c5463f0ee735fe123c0cf17f251462ec940d395a664b0a2ec9" gracePeriod=30 Oct 30 00:32:34.046789 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.046755 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f3e21ed-vcpck"] Oct 30 00:32:34.047373 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.047358 5001 cpu_manager.go:414] "containerMap: removing container" podUID="bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6" containerName="istio-proxy" Oct 30 00:32:34.047373 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.047373 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6" containerName="istio-proxy" Oct 30 00:32:34.047450 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.047394 5001 cpu_manager.go:414] "containerMap: removing container" podUID="bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6" containerName="istio-proxy" Oct 30 00:32:34.047450 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.047399 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6" containerName="istio-proxy" Oct 30 00:32:34.047450 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.047415 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1832bfed-6454-4368-bcd5-223c048fcc36" containerName="revoke-keys" Oct 30 00:32:34.047450 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.047420 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1832bfed-6454-4368-bcd5-223c048fcc36" containerName="revoke-keys" Oct 30 00:32:34.047450 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.047443 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f2f023ad-b552-4c5a-89bf-aac6755064f1" containerName="revoke-keys" Oct 30 00:32:34.047450 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.047447 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2f023ad-b552-4c5a-89bf-aac6755064f1" containerName="revoke-keys" Oct 30 00:32:34.047647 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.047537 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6" containerName="istio-proxy" Oct 30 00:32:34.047647 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.047547 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1832bfed-6454-4368-bcd5-223c048fcc36" containerName="revoke-keys" Oct 30 00:32:34.047647 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.047558 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2f023ad-b552-4c5a-89bf-aac6755064f1" containerName="revoke-keys" Oct 30 00:32:34.052257 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.052235 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3e21ed-vcpck" Oct 30 00:32:34.071871 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.071844 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:32:34.081555 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.081532 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:32:34.081640 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.081580 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:32:34.081640 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.081608 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:32:34.104117 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.104087 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f3e21ed-vcpck"] Oct 30 00:32:34.133754 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.133722 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cswp8\" (UniqueName: \"kubernetes.io/projected/c356c3ec-def9-47ab-ad8c-88620887acac-kube-api-access-cswp8\") pod \"maas-api-revoke-sub-7f3e21ed-vcpck\" (UID: \"c356c3ec-def9-47ab-ad8c-88620887acac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3e21ed-vcpck" Oct 30 00:32:34.133989 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.133967 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c356c3ec-def9-47ab-ad8c-88620887acac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f3e21ed-vcpck\" (UID: \"c356c3ec-def9-47ab-ad8c-88620887acac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3e21ed-vcpck" Oct 30 00:32:34.134114 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.134096 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c356c3ec-def9-47ab-ad8c-88620887acac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f3e21ed-vcpck\" (UID: \"c356c3ec-def9-47ab-ad8c-88620887acac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3e21ed-vcpck" Oct 30 00:32:34.235566 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.235533 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cswp8\" (UniqueName: \"kubernetes.io/projected/c356c3ec-def9-47ab-ad8c-88620887acac-kube-api-access-cswp8\") pod \"maas-api-revoke-sub-7f3e21ed-vcpck\" (UID: \"c356c3ec-def9-47ab-ad8c-88620887acac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3e21ed-vcpck" Oct 30 00:32:34.235774 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.235674 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c356c3ec-def9-47ab-ad8c-88620887acac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f3e21ed-vcpck\" (UID: \"c356c3ec-def9-47ab-ad8c-88620887acac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3e21ed-vcpck" Oct 30 00:32:34.235774 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.235741 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c356c3ec-def9-47ab-ad8c-88620887acac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f3e21ed-vcpck\" (UID: \"c356c3ec-def9-47ab-ad8c-88620887acac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3e21ed-vcpck" Oct 30 00:32:34.236425 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.236402 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c356c3ec-def9-47ab-ad8c-88620887acac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f3e21ed-vcpck\" (UID: \"c356c3ec-def9-47ab-ad8c-88620887acac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3e21ed-vcpck" Oct 30 00:32:34.238237 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.238212 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c356c3ec-def9-47ab-ad8c-88620887acac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f3e21ed-vcpck\" (UID: \"c356c3ec-def9-47ab-ad8c-88620887acac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3e21ed-vcpck" Oct 30 00:32:34.255781 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.255757 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cswp8\" (UniqueName: \"kubernetes.io/projected/c356c3ec-def9-47ab-ad8c-88620887acac-kube-api-access-cswp8\") pod \"maas-api-revoke-sub-7f3e21ed-vcpck\" (UID: \"c356c3ec-def9-47ab-ad8c-88620887acac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3e21ed-vcpck" Oct 30 00:32:34.366508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.366426 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3e21ed-vcpck" Oct 30 00:32:34.671129 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:32:34.671099 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc356c3ec_def9_47ab_ad8c_88620887acac.slice/crio-7e2a0dee9d848ba9c085fb469557af5cc7c64b11f22b55e33603f11a27b8335b WatchSource:0}: Error finding container 7e2a0dee9d848ba9c085fb469557af5cc7c64b11f22b55e33603f11a27b8335b: Status 404 returned error can't find the container with id 7e2a0dee9d848ba9c085fb469557af5cc7c64b11f22b55e33603f11a27b8335b Oct 30 00:32:34.671512 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.671493 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f3e21ed-vcpck"] Oct 30 00:32:34.933490 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.933460 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d034487-g9qjg"] Oct 30 00:32:34.934234 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.934218 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="bffa67e6-fcdf-4021-ab0d-9b0c2a5d09e6" containerName="istio-proxy" Oct 30 00:32:34.937869 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.937843 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d034487-g9qjg" Oct 30 00:32:34.945325 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.945300 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6xkb\" (UniqueName: \"kubernetes.io/projected/253eb22e-11d4-47c6-be47-40f024aff63d-kube-api-access-g6xkb\") pod \"maas-api-revoke-sub-5d034487-g9qjg\" (UID: \"253eb22e-11d4-47c6-be47-40f024aff63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d034487-g9qjg" Oct 30 00:32:34.945518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.945486 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253eb22e-11d4-47c6-be47-40f024aff63d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d034487-g9qjg\" (UID: \"253eb22e-11d4-47c6-be47-40f024aff63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d034487-g9qjg" Oct 30 00:32:34.945644 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.945566 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253eb22e-11d4-47c6-be47-40f024aff63d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d034487-g9qjg\" (UID: \"253eb22e-11d4-47c6-be47-40f024aff63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d034487-g9qjg" Oct 30 00:32:34.958149 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:34.958124 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d034487-g9qjg"] Oct 30 00:32:35.047342 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:35.047312 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6xkb\" (UniqueName: \"kubernetes.io/projected/253eb22e-11d4-47c6-be47-40f024aff63d-kube-api-access-g6xkb\") pod \"maas-api-revoke-sub-5d034487-g9qjg\" (UID: \"253eb22e-11d4-47c6-be47-40f024aff63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d034487-g9qjg" Oct 30 00:32:35.047740 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:35.047459 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253eb22e-11d4-47c6-be47-40f024aff63d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d034487-g9qjg\" (UID: \"253eb22e-11d4-47c6-be47-40f024aff63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d034487-g9qjg" Oct 30 00:32:35.047740 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:35.047524 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253eb22e-11d4-47c6-be47-40f024aff63d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d034487-g9qjg\" (UID: \"253eb22e-11d4-47c6-be47-40f024aff63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d034487-g9qjg" Oct 30 00:32:35.048154 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:35.048134 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253eb22e-11d4-47c6-be47-40f024aff63d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d034487-g9qjg\" (UID: \"253eb22e-11d4-47c6-be47-40f024aff63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d034487-g9qjg" Oct 30 00:32:35.049731 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:35.049715 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253eb22e-11d4-47c6-be47-40f024aff63d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d034487-g9qjg\" (UID: \"253eb22e-11d4-47c6-be47-40f024aff63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d034487-g9qjg" Oct 30 00:32:35.067991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:35.067968 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6xkb\" (UniqueName: \"kubernetes.io/projected/253eb22e-11d4-47c6-be47-40f024aff63d-kube-api-access-g6xkb\") pod \"maas-api-revoke-sub-5d034487-g9qjg\" (UID: \"253eb22e-11d4-47c6-be47-40f024aff63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d034487-g9qjg" Oct 30 00:32:35.252435 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:35.252359 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d034487-g9qjg" Oct 30 00:32:35.493220 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:35.493182 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3e21ed-vcpck" event={"ID":"c356c3ec-def9-47ab-ad8c-88620887acac","Type":"ContainerStarted","Data":"dad740d78220851e665c0f5a2febdc8c994ab915045a57478d0ad774a84e9cda"} Oct 30 00:32:35.493431 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:35.493253 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3e21ed-vcpck" event={"ID":"c356c3ec-def9-47ab-ad8c-88620887acac","Type":"ContainerStarted","Data":"7e2a0dee9d848ba9c085fb469557af5cc7c64b11f22b55e33603f11a27b8335b"} Oct 30 00:32:35.528647 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:35.528601 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3e21ed-vcpck" podStartSLOduration=2.528588408 podStartE2EDuration="2.528588408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:35.528075625 +0000 UTC m=+1670.733576826" watchObservedRunningTime="2026-10-30 00:32:35.528588408 +0000 UTC m=+1670.734089589" Oct 30 00:32:35.568990 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:32:35.568962 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod253eb22e_11d4_47c6_be47_40f024aff63d.slice/crio-44cb4634a8de5e30bb98cc8f6cd511518f4a94abc348668cd2bf75edc5cb1af2 WatchSource:0}: Error finding container 44cb4634a8de5e30bb98cc8f6cd511518f4a94abc348668cd2bf75edc5cb1af2: Status 404 returned error can't find the container with id 44cb4634a8de5e30bb98cc8f6cd511518f4a94abc348668cd2bf75edc5cb1af2 Oct 30 00:32:35.570586 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:35.570553 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d034487-g9qjg"] Oct 30 00:32:36.503388 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:36.503304 5001 generic.go:356] "Generic (PLEG): container finished" podID="c356c3ec-def9-47ab-ad8c-88620887acac" containerID="dad740d78220851e665c0f5a2febdc8c994ab915045a57478d0ad774a84e9cda" exitCode=0 Oct 30 00:32:36.503801 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:36.503407 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3e21ed-vcpck" event={"ID":"c356c3ec-def9-47ab-ad8c-88620887acac","Type":"ContainerDied","Data":"dad740d78220851e665c0f5a2febdc8c994ab915045a57478d0ad774a84e9cda"} Oct 30 00:32:36.505314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:36.505280 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d034487-g9qjg" event={"ID":"253eb22e-11d4-47c6-be47-40f024aff63d","Type":"ContainerStarted","Data":"cbff866fef072bd37e2287c0ceca83485cf5324e182f38fcd3fa97910f69b0d8"} Oct 30 00:32:36.505465 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:36.505321 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d034487-g9qjg" event={"ID":"253eb22e-11d4-47c6-be47-40f024aff63d","Type":"ContainerStarted","Data":"44cb4634a8de5e30bb98cc8f6cd511518f4a94abc348668cd2bf75edc5cb1af2"} Oct 30 00:32:36.576727 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:36.576671 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d034487-g9qjg" podStartSLOduration=2.57665461 podStartE2EDuration="2.57665461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:36.573308473 +0000 UTC m=+1671.778809666" watchObservedRunningTime="2026-10-30 00:32:36.57665461 +0000 UTC m=+1671.782155791" Oct 30 00:32:37.352591 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.352532 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:32:37.435351 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.435313 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-workload-certs\") pod \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " Oct 30 00:32:37.435538 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.435363 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istio-envoy\") pod \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " Oct 30 00:32:37.435538 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.435383 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istiod-ca-cert\") pod \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " Oct 30 00:32:37.435538 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.435444 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-workload-socket\") pod \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " Oct 30 00:32:37.435538 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.435477 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-kube-api-access-rs5fz\" (UniqueName: \"kubernetes.io/projected/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-kube-api-access-rs5fz\") pod \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " Oct 30 00:32:37.435817 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.435610 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istio-token\" (UniqueName: \"kubernetes.io/projected/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istio-token\") pod \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " Oct 30 00:32:37.435817 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.435699 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istio-podinfo\") pod \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " Oct 30 00:32:37.435817 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.435735 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-workload-certs" pod "bdd71cc9-c2f9-4924-ab67-4b5f03b08f90" (UID: "bdd71cc9-c2f9-4924-ab67-4b5f03b08f90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:37.435817 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.435788 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-workload-socket" pod "bdd71cc9-c2f9-4924-ab67-4b5f03b08f90" (UID: "bdd71cc9-c2f9-4924-ab67-4b5f03b08f90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:37.436068 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.435900 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istio-data\") pod \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " Oct 30 00:32:37.436068 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.435907 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istiod-ca-cert" pod "bdd71cc9-c2f9-4924-ab67-4b5f03b08f90" (UID: "bdd71cc9-c2f9-4924-ab67-4b5f03b08f90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:32:37.436068 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.436017 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-credential-socket\") pod \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\" (UID: \"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90\") " Oct 30 00:32:37.436263 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.436223 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istio-data" pod "bdd71cc9-c2f9-4924-ab67-4b5f03b08f90" (UID: "bdd71cc9-c2f9-4924-ab67-4b5f03b08f90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:37.436509 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.436464 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-credential-socket" pod "bdd71cc9-c2f9-4924-ab67-4b5f03b08f90" (UID: "bdd71cc9-c2f9-4924-ab67-4b5f03b08f90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:37.436874 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.436848 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:37.437013 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.436880 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:37.437013 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.436895 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:37.437013 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.436924 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:37.437013 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.436938 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:37.438426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.438402 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istio-podinfo" pod "bdd71cc9-c2f9-4924-ab67-4b5f03b08f90" (UID: "bdd71cc9-c2f9-4924-ab67-4b5f03b08f90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:32:37.438542 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.438523 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istio-token" pod "bdd71cc9-c2f9-4924-ab67-4b5f03b08f90" (UID: "bdd71cc9-c2f9-4924-ab67-4b5f03b08f90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:32:37.438787 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.438768 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istio-envoy" pod "bdd71cc9-c2f9-4924-ab67-4b5f03b08f90" (UID: "bdd71cc9-c2f9-4924-ab67-4b5f03b08f90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:37.439211 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.439192 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-kube-api-access-rs5fz" pod "bdd71cc9-c2f9-4924-ab67-4b5f03b08f90" (UID: "bdd71cc9-c2f9-4924-ab67-4b5f03b08f90"). InnerVolumeSpecName "kube-api-access-rs5fz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:32:37.515279 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.515243 5001 generic.go:356] "Generic (PLEG): container finished" podID="bdd71cc9-c2f9-4924-ab67-4b5f03b08f90" containerID="cf080347689293c5463f0ee735fe123c0cf17f251462ec940d395a664b0a2ec9" exitCode=0 Oct 30 00:32:37.515688 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.515286 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" Oct 30 00:32:37.515688 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.515325 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" event={"ID":"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90","Type":"ContainerDied","Data":"cf080347689293c5463f0ee735fe123c0cf17f251462ec940d395a664b0a2ec9"} Oct 30 00:32:37.515688 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.515385 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k" event={"ID":"bdd71cc9-c2f9-4924-ab67-4b5f03b08f90","Type":"ContainerDied","Data":"ec5a0e2581f5d15017d42ccee2e55240424ea671f5aa89d7174ef7c3aa84ea1e"} Oct 30 00:32:37.515688 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.515413 5001 scope.go:122] "RemoveContainer" containerID="cf080347689293c5463f0ee735fe123c0cf17f251462ec940d395a664b0a2ec9" Oct 30 00:32:37.517437 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.517413 5001 generic.go:356] "Generic (PLEG): container finished" podID="253eb22e-11d4-47c6-be47-40f024aff63d" containerID="cbff866fef072bd37e2287c0ceca83485cf5324e182f38fcd3fa97910f69b0d8" exitCode=0 Oct 30 00:32:37.517554 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.517500 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d034487-g9qjg" event={"ID":"253eb22e-11d4-47c6-be47-40f024aff63d","Type":"ContainerDied","Data":"cbff866fef072bd37e2287c0ceca83485cf5324e182f38fcd3fa97910f69b0d8"} Oct 30 00:32:37.530855 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.530838 5001 scope.go:122] "RemoveContainer" containerID="be7a44843a754b3e018a34b63f1c16f6efb066a59e0d101031e550dd7a7808fe" Oct 30 00:32:37.537843 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.537818 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:37.537843 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.537845 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs5fz\" (UniqueName: \"kubernetes.io/projected/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-kube-api-access-rs5fz\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:37.537997 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.537854 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:37.537997 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.537865 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:37.546951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.546891 5001 scope.go:122] "RemoveContainer" containerID="cf080347689293c5463f0ee735fe123c0cf17f251462ec940d395a664b0a2ec9" Oct 30 00:32:37.547603 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:32:37.547579 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf080347689293c5463f0ee735fe123c0cf17f251462ec940d395a664b0a2ec9\": container with ID starting with cf080347689293c5463f0ee735fe123c0cf17f251462ec940d395a664b0a2ec9 not found: ID does not exist" containerID="cf080347689293c5463f0ee735fe123c0cf17f251462ec940d395a664b0a2ec9" Oct 30 00:32:37.547703 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.547611 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf080347689293c5463f0ee735fe123c0cf17f251462ec940d395a664b0a2ec9"} err="failed to get container status \"cf080347689293c5463f0ee735fe123c0cf17f251462ec940d395a664b0a2ec9\": rpc error: code = NotFound desc = could not find container \"cf080347689293c5463f0ee735fe123c0cf17f251462ec940d395a664b0a2ec9\": container with ID starting with cf080347689293c5463f0ee735fe123c0cf17f251462ec940d395a664b0a2ec9 not found: ID does not exist" Oct 30 00:32:37.547703 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.547630 5001 scope.go:122] "RemoveContainer" containerID="be7a44843a754b3e018a34b63f1c16f6efb066a59e0d101031e550dd7a7808fe" Oct 30 00:32:37.547889 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:32:37.547874 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be7a44843a754b3e018a34b63f1c16f6efb066a59e0d101031e550dd7a7808fe\": container with ID starting with be7a44843a754b3e018a34b63f1c16f6efb066a59e0d101031e550dd7a7808fe not found: ID does not exist" containerID="be7a44843a754b3e018a34b63f1c16f6efb066a59e0d101031e550dd7a7808fe" Oct 30 00:32:37.547976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.547895 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be7a44843a754b3e018a34b63f1c16f6efb066a59e0d101031e550dd7a7808fe"} err="failed to get container status \"be7a44843a754b3e018a34b63f1c16f6efb066a59e0d101031e550dd7a7808fe\": rpc error: code = NotFound desc = could not find container \"be7a44843a754b3e018a34b63f1c16f6efb066a59e0d101031e550dd7a7808fe\": container with ID starting with be7a44843a754b3e018a34b63f1c16f6efb066a59e0d101031e550dd7a7808fe not found: ID does not exist" Oct 30 00:32:37.601489 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.601449 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k"] Oct 30 00:32:37.609883 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.609810 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-bc8b7d-openshift-default-95697684f-5b88k"] Oct 30 00:32:37.681839 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.681814 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3e21ed-vcpck" Oct 30 00:32:37.842687 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.842653 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c356c3ec-def9-47ab-ad8c-88620887acac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c356c3ec-def9-47ab-ad8c-88620887acac-maas-api-xks-ca\") pod \"c356c3ec-def9-47ab-ad8c-88620887acac\" (UID: \"c356c3ec-def9-47ab-ad8c-88620887acac\") " Oct 30 00:32:37.842939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.842898 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c356c3ec-def9-47ab-ad8c-88620887acac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c356c3ec-def9-47ab-ad8c-88620887acac-maas-api-openshift-ca\") pod \"c356c3ec-def9-47ab-ad8c-88620887acac\" (UID: \"c356c3ec-def9-47ab-ad8c-88620887acac\") " Oct 30 00:32:37.843013 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.843000 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c356c3ec-def9-47ab-ad8c-88620887acac-kube-api-access-cswp8\" (UniqueName: \"kubernetes.io/projected/c356c3ec-def9-47ab-ad8c-88620887acac-kube-api-access-cswp8\") pod \"c356c3ec-def9-47ab-ad8c-88620887acac\" (UID: \"c356c3ec-def9-47ab-ad8c-88620887acac\") " Oct 30 00:32:37.843282 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.843254 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c356c3ec-def9-47ab-ad8c-88620887acac-maas-api-openshift-ca" pod "c356c3ec-def9-47ab-ad8c-88620887acac" (UID: "c356c3ec-def9-47ab-ad8c-88620887acac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:32:37.843488 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.843471 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c356c3ec-def9-47ab-ad8c-88620887acac-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:37.844797 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.844779 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c356c3ec-def9-47ab-ad8c-88620887acac-maas-api-xks-ca" pod "c356c3ec-def9-47ab-ad8c-88620887acac" (UID: "c356c3ec-def9-47ab-ad8c-88620887acac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:32:37.844972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.844953 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c356c3ec-def9-47ab-ad8c-88620887acac-kube-api-access-cswp8" pod "c356c3ec-def9-47ab-ad8c-88620887acac" (UID: "c356c3ec-def9-47ab-ad8c-88620887acac"). InnerVolumeSpecName "kube-api-access-cswp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:32:37.944687 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.944616 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c356c3ec-def9-47ab-ad8c-88620887acac-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:37.944687 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:37.944640 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cswp8\" (UniqueName: \"kubernetes.io/projected/c356c3ec-def9-47ab-ad8c-88620887acac-kube-api-access-cswp8\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:38.035882 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:38.035849 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6"] Oct 30 00:32:38.036109 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:38.036085 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" podUID="48b4143b-d307-4714-bb2c-8704a330753a" containerName="istio-proxy" containerID="cri-o://0b2e80c59351f503e4626c57b772869beaa0e031ddb5d0c7ffeb8fde9cff2171" gracePeriod=30 Oct 30 00:32:38.040777 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:38.040752 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm"] Oct 30 00:32:38.040976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:38.040941 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" podUID="35bc1599-694f-4f89-9a8a-7667310b9b90" containerName="istio-proxy" containerID="cri-o://f9ca5bf35f7104390a99ff08f661910cc9f0e185ae680b4d385dc30643eef04f" gracePeriod=30 Oct 30 00:32:38.244772 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:38.244677 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:32:38.244772 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:38.244743 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" podUID="35bc1599-694f-4f89-9a8a-7667310b9b90" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:32:38.529656 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:38.529624 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3e21ed-vcpck" Oct 30 00:32:38.530113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:38.529666 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3e21ed-vcpck" event={"ID":"c356c3ec-def9-47ab-ad8c-88620887acac","Type":"ContainerDied","Data":"7e2a0dee9d848ba9c085fb469557af5cc7c64b11f22b55e33603f11a27b8335b"} Oct 30 00:32:38.530113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:38.529713 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e2a0dee9d848ba9c085fb469557af5cc7c64b11f22b55e33603f11a27b8335b" Oct 30 00:32:38.703376 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:38.703333 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d034487-g9qjg" Oct 30 00:32:38.861544 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:38.861463 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/253eb22e-11d4-47c6-be47-40f024aff63d-kube-api-access-g6xkb\" (UniqueName: \"kubernetes.io/projected/253eb22e-11d4-47c6-be47-40f024aff63d-kube-api-access-g6xkb\") pod \"253eb22e-11d4-47c6-be47-40f024aff63d\" (UID: \"253eb22e-11d4-47c6-be47-40f024aff63d\") " Oct 30 00:32:38.861544 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:38.861513 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/253eb22e-11d4-47c6-be47-40f024aff63d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253eb22e-11d4-47c6-be47-40f024aff63d-maas-api-xks-ca\") pod \"253eb22e-11d4-47c6-be47-40f024aff63d\" (UID: \"253eb22e-11d4-47c6-be47-40f024aff63d\") " Oct 30 00:32:38.861812 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:38.861556 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/253eb22e-11d4-47c6-be47-40f024aff63d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253eb22e-11d4-47c6-be47-40f024aff63d-maas-api-openshift-ca\") pod \"253eb22e-11d4-47c6-be47-40f024aff63d\" (UID: \"253eb22e-11d4-47c6-be47-40f024aff63d\") " Oct 30 00:32:38.861993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:38.861968 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/253eb22e-11d4-47c6-be47-40f024aff63d-maas-api-openshift-ca" pod "253eb22e-11d4-47c6-be47-40f024aff63d" (UID: "253eb22e-11d4-47c6-be47-40f024aff63d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:32:38.862400 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:38.862379 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253eb22e-11d4-47c6-be47-40f024aff63d-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:38.863953 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:38.863927 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/253eb22e-11d4-47c6-be47-40f024aff63d-maas-api-xks-ca" pod "253eb22e-11d4-47c6-be47-40f024aff63d" (UID: "253eb22e-11d4-47c6-be47-40f024aff63d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:32:38.864042 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:38.863975 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/253eb22e-11d4-47c6-be47-40f024aff63d-kube-api-access-g6xkb" pod "253eb22e-11d4-47c6-be47-40f024aff63d" (UID: "253eb22e-11d4-47c6-be47-40f024aff63d"). InnerVolumeSpecName "kube-api-access-g6xkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:32:38.919138 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:38.919109 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdd71cc9-c2f9-4924-ab67-4b5f03b08f90" path="/var/lib/kubelet/pods/bdd71cc9-c2f9-4924-ab67-4b5f03b08f90/volumes" Oct 30 00:32:38.963591 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:38.963558 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6xkb\" (UniqueName: \"kubernetes.io/projected/253eb22e-11d4-47c6-be47-40f024aff63d-kube-api-access-g6xkb\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:38.963591 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:38.963591 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253eb22e-11d4-47c6-be47-40f024aff63d-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:39.539254 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:39.539225 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d034487-g9qjg" Oct 30 00:32:39.539254 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:39.539236 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d034487-g9qjg" event={"ID":"253eb22e-11d4-47c6-be47-40f024aff63d","Type":"ContainerDied","Data":"44cb4634a8de5e30bb98cc8f6cd511518f4a94abc348668cd2bf75edc5cb1af2"} Oct 30 00:32:39.539740 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:39.539326 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44cb4634a8de5e30bb98cc8f6cd511518f4a94abc348668cd2bf75edc5cb1af2" Oct 30 00:32:40.173840 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:40.173808 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e7fdaac-cp4qd"] Oct 30 00:32:40.174415 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:40.174400 5001 cpu_manager.go:414] "containerMap: removing container" podUID="253eb22e-11d4-47c6-be47-40f024aff63d" containerName="revoke-keys" Oct 30 00:32:40.174478 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:40.174416 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="253eb22e-11d4-47c6-be47-40f024aff63d" containerName="revoke-keys" Oct 30 00:32:40.174478 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:40.174436 5001 cpu_manager.go:414] "containerMap: removing container" podUID="bdd71cc9-c2f9-4924-ab67-4b5f03b08f90" containerName="istio-proxy" Oct 30 00:32:40.174478 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:40.174441 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdd71cc9-c2f9-4924-ab67-4b5f03b08f90" containerName="istio-proxy" Oct 30 00:32:40.174478 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:40.174449 5001 cpu_manager.go:414] "containerMap: removing container" podUID="c356c3ec-def9-47ab-ad8c-88620887acac" containerName="revoke-keys" Oct 30 00:32:40.174478 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:40.174454 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c356c3ec-def9-47ab-ad8c-88620887acac" containerName="revoke-keys" Oct 30 00:32:40.174693 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:40.174484 5001 cpu_manager.go:414] "containerMap: removing container" podUID="bdd71cc9-c2f9-4924-ab67-4b5f03b08f90" containerName="istio-proxy" Oct 30 00:32:40.174693 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:40.174489 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdd71cc9-c2f9-4924-ab67-4b5f03b08f90" containerName="istio-proxy" Oct 30 00:32:40.174693 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:40.174593 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdd71cc9-c2f9-4924-ab67-4b5f03b08f90" containerName="istio-proxy" Oct 30 00:32:40.174693 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:40.174603 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="253eb22e-11d4-47c6-be47-40f024aff63d" containerName="revoke-keys" Oct 30 00:32:40.174693 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:40.174613 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdd71cc9-c2f9-4924-ab67-4b5f03b08f90" containerName="istio-proxy" Oct 30 00:32:40.174693 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:40.174618 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c356c3ec-def9-47ab-ad8c-88620887acac" containerName="revoke-keys" Oct 30 00:32:40.179345 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:40.179325 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7fdaac-cp4qd" Oct 30 00:32:40.193205 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:40.193185 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:32:40.204187 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:40.203288 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:32:40.204187 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:40.203294 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:32:40.204187 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:40.203399 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:32:40.211610 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:40.211586 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e7fdaac-cp4qd"] Oct 30 00:32:40.282473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:40.282436 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt69w\" (UniqueName: \"kubernetes.io/projected/779d2d08-dbfe-4dd0-896f-25213ab7c12f-kube-api-access-dt69w\") pod \"maas-api-revoke-sub-2e7fdaac-cp4qd\" (UID: \"779d2d08-dbfe-4dd0-896f-25213ab7c12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7fdaac-cp4qd" Oct 30 00:32:40.282644 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:40.282583 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/779d2d08-dbfe-4dd0-896f-25213ab7c12f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e7fdaac-cp4qd\" (UID: \"779d2d08-dbfe-4dd0-896f-25213ab7c12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7fdaac-cp4qd" Oct 30 00:32:40.282644 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:40.282619 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/779d2d08-dbfe-4dd0-896f-25213ab7c12f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e7fdaac-cp4qd\" (UID: \"779d2d08-dbfe-4dd0-896f-25213ab7c12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7fdaac-cp4qd" Oct 30 00:32:40.383432 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:40.383404 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/779d2d08-dbfe-4dd0-896f-25213ab7c12f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e7fdaac-cp4qd\" (UID: \"779d2d08-dbfe-4dd0-896f-25213ab7c12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7fdaac-cp4qd" Oct 30 00:32:40.383625 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:40.383440 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/779d2d08-dbfe-4dd0-896f-25213ab7c12f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e7fdaac-cp4qd\" (UID: \"779d2d08-dbfe-4dd0-896f-25213ab7c12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7fdaac-cp4qd" Oct 30 00:32:40.383625 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:40.383576 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt69w\" (UniqueName: \"kubernetes.io/projected/779d2d08-dbfe-4dd0-896f-25213ab7c12f-kube-api-access-dt69w\") pod \"maas-api-revoke-sub-2e7fdaac-cp4qd\" (UID: \"779d2d08-dbfe-4dd0-896f-25213ab7c12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7fdaac-cp4qd" Oct 30 00:32:40.384145 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:40.384122 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/779d2d08-dbfe-4dd0-896f-25213ab7c12f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e7fdaac-cp4qd\" (UID: \"779d2d08-dbfe-4dd0-896f-25213ab7c12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7fdaac-cp4qd" Oct 30 00:32:40.385927 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:40.385896 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/779d2d08-dbfe-4dd0-896f-25213ab7c12f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e7fdaac-cp4qd\" (UID: \"779d2d08-dbfe-4dd0-896f-25213ab7c12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7fdaac-cp4qd" Oct 30 00:32:40.407192 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:40.407161 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt69w\" (UniqueName: \"kubernetes.io/projected/779d2d08-dbfe-4dd0-896f-25213ab7c12f-kube-api-access-dt69w\") pod \"maas-api-revoke-sub-2e7fdaac-cp4qd\" (UID: \"779d2d08-dbfe-4dd0-896f-25213ab7c12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7fdaac-cp4qd" Oct 30 00:32:40.493836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:40.493770 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7fdaac-cp4qd" Oct 30 00:32:40.799309 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:32:40.799276 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod779d2d08_dbfe_4dd0_896f_25213ab7c12f.slice/crio-2c03ce30bcf8bd929ce3afc326e1ab8f981d464b62451ac9a1452b6e3831dd80 WatchSource:0}: Error finding container 2c03ce30bcf8bd929ce3afc326e1ab8f981d464b62451ac9a1452b6e3831dd80: Status 404 returned error can't find the container with id 2c03ce30bcf8bd929ce3afc326e1ab8f981d464b62451ac9a1452b6e3831dd80 Oct 30 00:32:40.801856 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:40.801828 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e7fdaac-cp4qd"] Oct 30 00:32:41.561228 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:41.561194 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7fdaac-cp4qd" event={"ID":"779d2d08-dbfe-4dd0-896f-25213ab7c12f","Type":"ContainerStarted","Data":"2b12a2d48fde9fbd26a91142b1c666984f10c21009c12fc7d5282f35b534fe29"} Oct 30 00:32:41.561418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:41.561243 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7fdaac-cp4qd" event={"ID":"779d2d08-dbfe-4dd0-896f-25213ab7c12f","Type":"ContainerStarted","Data":"2c03ce30bcf8bd929ce3afc326e1ab8f981d464b62451ac9a1452b6e3831dd80"} Oct 30 00:32:41.600469 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:41.600417 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7fdaac-cp4qd" podStartSLOduration=1.600402811 podStartE2EDuration="1.600402811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:41.596550733 +0000 UTC m=+1676.802051965" watchObservedRunningTime="2026-10-30 00:32:41.600402811 +0000 UTC m=+1676.805903992" Oct 30 00:32:42.615935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:42.614859 5001 generic.go:356] "Generic (PLEG): container finished" podID="779d2d08-dbfe-4dd0-896f-25213ab7c12f" containerID="2b12a2d48fde9fbd26a91142b1c666984f10c21009c12fc7d5282f35b534fe29" exitCode=0 Oct 30 00:32:42.615935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:42.615179 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7fdaac-cp4qd" event={"ID":"779d2d08-dbfe-4dd0-896f-25213ab7c12f","Type":"ContainerDied","Data":"2b12a2d48fde9fbd26a91142b1c666984f10c21009c12fc7d5282f35b534fe29"} Oct 30 00:32:43.017778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.017747 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn"] Oct 30 00:32:43.022098 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.022070 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:32:43.035640 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.035614 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-c1eeb6-openshift-default-dockercfg-dj7mw\"" Oct 30 00:32:43.052575 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.052269 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn"] Oct 30 00:32:43.239939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.228419 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4deafe88-91e1-4878-9203-e28de81ce261-istio-podinfo\") pod \"e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:32:43.239939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.228546 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-istio-data\") pod \"e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:32:43.239939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.228623 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-credential-socket\") pod \"e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:32:43.239939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.228669 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-workload-certs\") pod \"e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:32:43.239939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.228691 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4deafe88-91e1-4878-9203-e28de81ce261-istiod-ca-cert\") pod \"e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:32:43.239939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.228725 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-workload-socket\") pod \"e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:32:43.239939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.228772 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j645z\" (UniqueName: \"kubernetes.io/projected/4deafe88-91e1-4878-9203-e28de81ce261-kube-api-access-j645z\") pod \"e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:32:43.239939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.228855 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4deafe88-91e1-4878-9203-e28de81ce261-istio-token\") pod \"e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:32:43.239939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.228948 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-istio-envoy\") pod \"e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:32:43.337942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.333150 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-workload-socket\") pod \"e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:32:43.337942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.333232 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j645z\" (UniqueName: \"kubernetes.io/projected/4deafe88-91e1-4878-9203-e28de81ce261-kube-api-access-j645z\") pod \"e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:32:43.337942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.333316 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4deafe88-91e1-4878-9203-e28de81ce261-istio-token\") pod \"e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:32:43.337942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.333393 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-istio-envoy\") pod \"e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:32:43.337942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.333485 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4deafe88-91e1-4878-9203-e28de81ce261-istio-podinfo\") pod \"e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:32:43.337942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.333587 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-istio-data\") pod \"e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:32:43.337942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.333664 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-credential-socket\") pod \"e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:32:43.337942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.333710 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-workload-certs\") pod \"e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:32:43.337942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.333735 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4deafe88-91e1-4878-9203-e28de81ce261-istiod-ca-cert\") pod \"e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:32:43.337942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.334619 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4deafe88-91e1-4878-9203-e28de81ce261-istiod-ca-cert\") pod \"e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:32:43.347583 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.339344 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-credential-socket\") pod \"e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:32:43.347583 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.339594 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-istio-data\") pod \"e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:32:43.347583 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.339823 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-workload-certs\") pod \"e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:32:43.347583 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.340391 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-workload-socket\") pod \"e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:32:43.378657 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.378620 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j645z\" (UniqueName: \"kubernetes.io/projected/4deafe88-91e1-4878-9203-e28de81ce261-kube-api-access-j645z\") pod \"e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:32:43.379655 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.379338 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-istio-envoy\") pod \"e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:32:43.380150 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.380118 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4deafe88-91e1-4878-9203-e28de81ce261-istio-token\") pod \"e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:32:43.398308 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.381025 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4deafe88-91e1-4878-9203-e28de81ce261-istio-podinfo\") pod \"e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:32:43.632720 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.632684 5001 generic.go:356] "Generic (PLEG): container finished" podID="35bc1599-694f-4f89-9a8a-7667310b9b90" containerID="f9ca5bf35f7104390a99ff08f661910cc9f0e185ae680b4d385dc30643eef04f" exitCode=0 Oct 30 00:32:43.633208 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.632763 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" event={"ID":"35bc1599-694f-4f89-9a8a-7667310b9b90","Type":"ContainerDied","Data":"f9ca5bf35f7104390a99ff08f661910cc9f0e185ae680b4d385dc30643eef04f"} Oct 30 00:32:43.633208 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.632883 5001 scope.go:122] "RemoveContainer" containerID="f729a1e075e62ccca8b57280d27c2a18975fbeaf902c4c76fae53e6e51fd0d2b" Oct 30 00:32:43.635723 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.635274 5001 generic.go:356] "Generic (PLEG): container finished" podID="48b4143b-d307-4714-bb2c-8704a330753a" containerID="0b2e80c59351f503e4626c57b772869beaa0e031ddb5d0c7ffeb8fde9cff2171" exitCode=0 Oct 30 00:32:43.635723 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.635344 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" event={"ID":"48b4143b-d307-4714-bb2c-8704a330753a","Type":"ContainerDied","Data":"0b2e80c59351f503e4626c57b772869beaa0e031ddb5d0c7ffeb8fde9cff2171"} Oct 30 00:32:43.650345 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.650316 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:32:43.770018 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.769986 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:32:43.773510 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.773485 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:32:43.897603 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.897573 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7fdaac-cp4qd" Oct 30 00:32:43.952978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.952945 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35bc1599-694f-4f89-9a8a-7667310b9b90-kube-api-access-v4vwd\" (UniqueName: \"kubernetes.io/projected/35bc1599-694f-4f89-9a8a-7667310b9b90-kube-api-access-v4vwd\") pod \"35bc1599-694f-4f89-9a8a-7667310b9b90\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " Oct 30 00:32:43.953183 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.953006 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-istio-envoy\") pod \"48b4143b-d307-4714-bb2c-8704a330753a\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " Oct 30 00:32:43.953183 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.953036 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48b4143b-d307-4714-bb2c-8704a330753a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b4143b-d307-4714-bb2c-8704a330753a-istio-podinfo\") pod \"48b4143b-d307-4714-bb2c-8704a330753a\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " Oct 30 00:32:43.953183 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.953058 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48b4143b-d307-4714-bb2c-8704a330753a-istio-token\" (UniqueName: \"kubernetes.io/projected/48b4143b-d307-4714-bb2c-8704a330753a-istio-token\") pod \"48b4143b-d307-4714-bb2c-8704a330753a\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " Oct 30 00:32:43.953183 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.953100 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-workload-socket\") pod \"48b4143b-d307-4714-bb2c-8704a330753a\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " Oct 30 00:32:43.953183 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.953135 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-credential-socket\") pod \"48b4143b-d307-4714-bb2c-8704a330753a\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " Oct 30 00:32:43.953183 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.953186 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-credential-socket\") pod \"35bc1599-694f-4f89-9a8a-7667310b9b90\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " Oct 30 00:32:43.953545 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.953216 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35bc1599-694f-4f89-9a8a-7667310b9b90-istio-token\" (UniqueName: \"kubernetes.io/projected/35bc1599-694f-4f89-9a8a-7667310b9b90-istio-token\") pod \"35bc1599-694f-4f89-9a8a-7667310b9b90\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " Oct 30 00:32:43.953545 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.953232 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35bc1599-694f-4f89-9a8a-7667310b9b90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35bc1599-694f-4f89-9a8a-7667310b9b90-istio-podinfo\") pod \"35bc1599-694f-4f89-9a8a-7667310b9b90\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " Oct 30 00:32:43.953545 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.953250 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48b4143b-d307-4714-bb2c-8704a330753a-kube-api-access-sqjg2\" (UniqueName: \"kubernetes.io/projected/48b4143b-d307-4714-bb2c-8704a330753a-kube-api-access-sqjg2\") pod \"48b4143b-d307-4714-bb2c-8704a330753a\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " Oct 30 00:32:43.953545 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.953283 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-istio-data\") pod \"48b4143b-d307-4714-bb2c-8704a330753a\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " Oct 30 00:32:43.953545 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.953305 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-istio-data\") pod \"35bc1599-694f-4f89-9a8a-7667310b9b90\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " Oct 30 00:32:43.953545 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.953369 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-workload-certs\") pod \"48b4143b-d307-4714-bb2c-8704a330753a\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " Oct 30 00:32:43.953545 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.953414 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35bc1599-694f-4f89-9a8a-7667310b9b90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35bc1599-694f-4f89-9a8a-7667310b9b90-istiod-ca-cert\") pod \"35bc1599-694f-4f89-9a8a-7667310b9b90\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " Oct 30 00:32:43.953545 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.953447 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-workload-socket\") pod \"35bc1599-694f-4f89-9a8a-7667310b9b90\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " Oct 30 00:32:43.954013 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.953557 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48b4143b-d307-4714-bb2c-8704a330753a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b4143b-d307-4714-bb2c-8704a330753a-istiod-ca-cert\") pod \"48b4143b-d307-4714-bb2c-8704a330753a\" (UID: \"48b4143b-d307-4714-bb2c-8704a330753a\") " Oct 30 00:32:43.954013 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.953565 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-credential-socket" pod "48b4143b-d307-4714-bb2c-8704a330753a" (UID: "48b4143b-d307-4714-bb2c-8704a330753a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:43.954013 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.953586 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-workload-certs\") pod \"35bc1599-694f-4f89-9a8a-7667310b9b90\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " Oct 30 00:32:43.954013 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.953700 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-istio-envoy\") pod \"35bc1599-694f-4f89-9a8a-7667310b9b90\" (UID: \"35bc1599-694f-4f89-9a8a-7667310b9b90\") " Oct 30 00:32:43.954553 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.954451 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:43.954676 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.954585 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-istio-data" pod "35bc1599-694f-4f89-9a8a-7667310b9b90" (UID: "35bc1599-694f-4f89-9a8a-7667310b9b90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:43.955025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.954799 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-istio-data" pod "48b4143b-d307-4714-bb2c-8704a330753a" (UID: "48b4143b-d307-4714-bb2c-8704a330753a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:43.955157 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.955134 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-workload-socket" pod "48b4143b-d307-4714-bb2c-8704a330753a" (UID: "48b4143b-d307-4714-bb2c-8704a330753a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:43.955404 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.955381 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-credential-socket" pod "35bc1599-694f-4f89-9a8a-7667310b9b90" (UID: "35bc1599-694f-4f89-9a8a-7667310b9b90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:43.955589 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.955568 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-workload-socket" pod "35bc1599-694f-4f89-9a8a-7667310b9b90" (UID: "35bc1599-694f-4f89-9a8a-7667310b9b90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:43.955780 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.955758 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-workload-certs" pod "48b4143b-d307-4714-bb2c-8704a330753a" (UID: "48b4143b-d307-4714-bb2c-8704a330753a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:43.956250 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.956020 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35bc1599-694f-4f89-9a8a-7667310b9b90-istiod-ca-cert" pod "35bc1599-694f-4f89-9a8a-7667310b9b90" (UID: "35bc1599-694f-4f89-9a8a-7667310b9b90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:32:43.956335 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.956247 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-istio-envoy" pod "48b4143b-d307-4714-bb2c-8704a330753a" (UID: "48b4143b-d307-4714-bb2c-8704a330753a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:43.956600 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.956574 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35bc1599-694f-4f89-9a8a-7667310b9b90-kube-api-access-v4vwd" pod "35bc1599-694f-4f89-9a8a-7667310b9b90" (UID: "35bc1599-694f-4f89-9a8a-7667310b9b90"). InnerVolumeSpecName "kube-api-access-v4vwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:32:43.956679 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.956607 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48b4143b-d307-4714-bb2c-8704a330753a-istiod-ca-cert" pod "48b4143b-d307-4714-bb2c-8704a330753a" (UID: "48b4143b-d307-4714-bb2c-8704a330753a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:32:43.956829 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.956808 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-workload-certs" pod "35bc1599-694f-4f89-9a8a-7667310b9b90" (UID: "35bc1599-694f-4f89-9a8a-7667310b9b90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:43.957003 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.956955 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-istio-envoy" pod "35bc1599-694f-4f89-9a8a-7667310b9b90" (UID: "35bc1599-694f-4f89-9a8a-7667310b9b90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:43.957003 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.956967 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48b4143b-d307-4714-bb2c-8704a330753a-kube-api-access-sqjg2" pod "48b4143b-d307-4714-bb2c-8704a330753a" (UID: "48b4143b-d307-4714-bb2c-8704a330753a"). InnerVolumeSpecName "kube-api-access-sqjg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:32:43.957796 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.957762 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48b4143b-d307-4714-bb2c-8704a330753a-istio-podinfo" pod "48b4143b-d307-4714-bb2c-8704a330753a" (UID: "48b4143b-d307-4714-bb2c-8704a330753a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:32:43.959332 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.959303 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48b4143b-d307-4714-bb2c-8704a330753a-istio-token" pod "48b4143b-d307-4714-bb2c-8704a330753a" (UID: "48b4143b-d307-4714-bb2c-8704a330753a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:32:43.962246 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.962222 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35bc1599-694f-4f89-9a8a-7667310b9b90-istio-podinfo" pod "35bc1599-694f-4f89-9a8a-7667310b9b90" (UID: "35bc1599-694f-4f89-9a8a-7667310b9b90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:32:43.963820 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:43.963793 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35bc1599-694f-4f89-9a8a-7667310b9b90-istio-token" pod "35bc1599-694f-4f89-9a8a-7667310b9b90" (UID: "35bc1599-694f-4f89-9a8a-7667310b9b90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:32:44.055139 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.041679 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn"] Oct 30 00:32:44.057943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.055835 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/779d2d08-dbfe-4dd0-896f-25213ab7c12f-kube-api-access-dt69w\" (UniqueName: \"kubernetes.io/projected/779d2d08-dbfe-4dd0-896f-25213ab7c12f-kube-api-access-dt69w\") pod \"779d2d08-dbfe-4dd0-896f-25213ab7c12f\" (UID: \"779d2d08-dbfe-4dd0-896f-25213ab7c12f\") " Oct 30 00:32:44.057943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.055888 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/779d2d08-dbfe-4dd0-896f-25213ab7c12f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/779d2d08-dbfe-4dd0-896f-25213ab7c12f-maas-api-xks-ca\") pod \"779d2d08-dbfe-4dd0-896f-25213ab7c12f\" (UID: \"779d2d08-dbfe-4dd0-896f-25213ab7c12f\") " Oct 30 00:32:44.057943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.057108 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/779d2d08-dbfe-4dd0-896f-25213ab7c12f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/779d2d08-dbfe-4dd0-896f-25213ab7c12f-maas-api-openshift-ca\") pod \"779d2d08-dbfe-4dd0-896f-25213ab7c12f\" (UID: \"779d2d08-dbfe-4dd0-896f-25213ab7c12f\") " Oct 30 00:32:44.057943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.057866 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:44.057943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.057888 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b4143b-d307-4714-bb2c-8704a330753a-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:44.069986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.057902 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48b4143b-d307-4714-bb2c-8704a330753a-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:44.069986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.060491 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:44.069986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.060509 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:44.069986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.060527 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35bc1599-694f-4f89-9a8a-7667310b9b90-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:44.069986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.060544 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35bc1599-694f-4f89-9a8a-7667310b9b90-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:44.069986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.060556 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqjg2\" (UniqueName: \"kubernetes.io/projected/48b4143b-d307-4714-bb2c-8704a330753a-kube-api-access-sqjg2\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:44.069986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.060571 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:44.069986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.060584 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:44.069986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.060596 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b4143b-d307-4714-bb2c-8704a330753a-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:44.069986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.060609 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35bc1599-694f-4f89-9a8a-7667310b9b90-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:44.069986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.060622 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:44.069986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.060635 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b4143b-d307-4714-bb2c-8704a330753a-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:44.069986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.060649 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:44.069986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.060662 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35bc1599-694f-4f89-9a8a-7667310b9b90-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:44.069986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.060675 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4vwd\" (UniqueName: \"kubernetes.io/projected/35bc1599-694f-4f89-9a8a-7667310b9b90-kube-api-access-v4vwd\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:44.069986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.058817 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/779d2d08-dbfe-4dd0-896f-25213ab7c12f-maas-api-openshift-ca" pod "779d2d08-dbfe-4dd0-896f-25213ab7c12f" (UID: "779d2d08-dbfe-4dd0-896f-25213ab7c12f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:32:44.069986 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:32:44.063080 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4deafe88_91e1_4878_9203_e28de81ce261.slice/crio-78828a98e504181c16613517d0590c654945b2f8b087c17c25ab476fc8c46f89 WatchSource:0}: Error finding container 78828a98e504181c16613517d0590c654945b2f8b087c17c25ab476fc8c46f89: Status 404 returned error can't find the container with id 78828a98e504181c16613517d0590c654945b2f8b087c17c25ab476fc8c46f89 Oct 30 00:32:44.069986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.066287 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/779d2d08-dbfe-4dd0-896f-25213ab7c12f-kube-api-access-dt69w" pod "779d2d08-dbfe-4dd0-896f-25213ab7c12f" (UID: "779d2d08-dbfe-4dd0-896f-25213ab7c12f"). InnerVolumeSpecName "kube-api-access-dt69w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:32:44.069986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.069197 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/779d2d08-dbfe-4dd0-896f-25213ab7c12f-maas-api-xks-ca" pod "779d2d08-dbfe-4dd0-896f-25213ab7c12f" (UID: "779d2d08-dbfe-4dd0-896f-25213ab7c12f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:32:44.095637 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.088773 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:32:44.095637 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.088866 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:32:44.095637 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.088959 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:32:44.162268 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.162235 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/779d2d08-dbfe-4dd0-896f-25213ab7c12f-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:44.162403 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.162272 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt69w\" (UniqueName: \"kubernetes.io/projected/779d2d08-dbfe-4dd0-896f-25213ab7c12f-kube-api-access-dt69w\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:44.162403 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.162289 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/779d2d08-dbfe-4dd0-896f-25213ab7c12f-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:44.646885 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.646856 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7fdaac-cp4qd" Oct 30 00:32:44.647409 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.646889 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7fdaac-cp4qd" event={"ID":"779d2d08-dbfe-4dd0-896f-25213ab7c12f","Type":"ContainerDied","Data":"2c03ce30bcf8bd929ce3afc326e1ab8f981d464b62451ac9a1452b6e3831dd80"} Oct 30 00:32:44.647409 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.646951 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c03ce30bcf8bd929ce3afc326e1ab8f981d464b62451ac9a1452b6e3831dd80" Oct 30 00:32:44.649031 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.649003 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" event={"ID":"4deafe88-91e1-4878-9203-e28de81ce261","Type":"ContainerStarted","Data":"933325317e94c64b0c6c337585297e652db8cab8364924dd9ede8a26aa107a3d"} Oct 30 00:32:44.649196 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.649047 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" event={"ID":"4deafe88-91e1-4878-9203-e28de81ce261","Type":"ContainerStarted","Data":"78828a98e504181c16613517d0590c654945b2f8b087c17c25ab476fc8c46f89"} Oct 30 00:32:44.650587 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.650554 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:32:44.650854 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.650839 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" Oct 30 00:32:44.650948 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.650893 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm" event={"ID":"35bc1599-694f-4f89-9a8a-7667310b9b90","Type":"ContainerDied","Data":"0c6d45180e45a143c757b0815ba0da9a85a78cbb8ce6a40c5961e3ef06480df5"} Oct 30 00:32:44.651025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.650975 5001 scope.go:122] "RemoveContainer" containerID="f9ca5bf35f7104390a99ff08f661910cc9f0e185ae680b4d385dc30643eef04f" Oct 30 00:32:44.652769 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.652750 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" Oct 30 00:32:44.652894 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.652837 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6" event={"ID":"48b4143b-d307-4714-bb2c-8704a330753a","Type":"ContainerDied","Data":"22be6ff189790b36c4ac0a28e98f7ec46624884b420163040505927de24bde7d"} Oct 30 00:32:44.670576 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.670387 5001 scope.go:122] "RemoveContainer" containerID="0b2e80c59351f503e4626c57b772869beaa0e031ddb5d0c7ffeb8fde9cff2171" Oct 30 00:32:44.690238 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.690187 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" podStartSLOduration=2.690171754 podStartE2EDuration="2.690171754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:44.688186972 +0000 UTC m=+1679.893688164" watchObservedRunningTime="2026-10-30 00:32:44.690171754 +0000 UTC m=+1679.895672945" Oct 30 00:32:44.723682 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.723646 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6"] Oct 30 00:32:44.728680 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.728652 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-q8gj6"] Oct 30 00:32:44.757943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.757894 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm"] Oct 30 00:32:44.763699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.763673 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5sdjm"] Oct 30 00:32:44.947225 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.947144 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35bc1599-694f-4f89-9a8a-7667310b9b90" path="/var/lib/kubelet/pods/35bc1599-694f-4f89-9a8a-7667310b9b90/volumes" Oct 30 00:32:44.947950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:44.947926 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48b4143b-d307-4714-bb2c-8704a330753a" path="/var/lib/kubelet/pods/48b4143b-d307-4714-bb2c-8704a330753a/volumes" Oct 30 00:32:45.703826 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:45.703789 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:32:45.829200 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:45.829162 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46"] Oct 30 00:32:45.830047 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:45.830021 5001 cpu_manager.go:414] "containerMap: removing container" podUID="35bc1599-694f-4f89-9a8a-7667310b9b90" containerName="istio-proxy" Oct 30 00:32:45.830047 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:45.830044 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35bc1599-694f-4f89-9a8a-7667310b9b90" containerName="istio-proxy" Oct 30 00:32:45.830258 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:45.830105 5001 cpu_manager.go:414] "containerMap: removing container" podUID="48b4143b-d307-4714-bb2c-8704a330753a" containerName="istio-proxy" Oct 30 00:32:45.830258 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:45.830113 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48b4143b-d307-4714-bb2c-8704a330753a" containerName="istio-proxy" Oct 30 00:32:45.830258 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:45.830133 5001 cpu_manager.go:414] "containerMap: removing container" podUID="779d2d08-dbfe-4dd0-896f-25213ab7c12f" containerName="revoke-keys" Oct 30 00:32:45.830258 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:45.830140 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="779d2d08-dbfe-4dd0-896f-25213ab7c12f" containerName="revoke-keys" Oct 30 00:32:45.830463 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:45.830305 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="48b4143b-d307-4714-bb2c-8704a330753a" containerName="istio-proxy" Oct 30 00:32:45.830463 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:45.830318 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="779d2d08-dbfe-4dd0-896f-25213ab7c12f" containerName="revoke-keys" Oct 30 00:32:45.830463 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:45.830333 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="35bc1599-694f-4f89-9a8a-7667310b9b90" containerName="istio-proxy" Oct 30 00:32:45.830463 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:45.830347 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="35bc1599-694f-4f89-9a8a-7667310b9b90" containerName="istio-proxy" Oct 30 00:32:45.835567 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:45.835543 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:32:45.863006 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:45.850331 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-29f3ca-openshift-default-dockercfg-fd2mk\"" Oct 30 00:32:45.863006 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:45.858939 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46"] Oct 30 00:32:45.902586 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:45.902540 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istio-token\") pod \"e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:32:45.902757 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:45.902694 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istio-envoy\") pod \"e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:32:45.902827 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:45.902753 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-workload-certs\") pod \"e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:32:45.902827 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:45.902782 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69pdb\" (UniqueName: \"kubernetes.io/projected/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-kube-api-access-69pdb\") pod \"e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:32:45.903025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:45.902827 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istio-data\") pod \"e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:32:45.903025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:45.902942 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-credential-socket\") pod \"e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:32:45.903025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:45.902977 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istio-podinfo\") pod \"e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:32:45.903196 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:45.903121 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istiod-ca-cert\") pod \"e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:32:45.903196 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:45.903167 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-workload-socket\") pod \"e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:32:46.018572 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:46.018520 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istio-token\") pod \"e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:32:46.018926 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:46.018889 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istio-envoy\") pod \"e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:32:46.019081 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:46.019064 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-workload-certs\") pod \"e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:32:46.019210 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:46.019192 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69pdb\" (UniqueName: \"kubernetes.io/projected/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-kube-api-access-69pdb\") pod \"e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:32:46.019353 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:46.019336 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istio-data\") pod \"e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:32:46.071166 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:46.067184 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-credential-socket\") pod \"e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:32:46.071166 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:46.067289 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istio-podinfo\") pod \"e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:32:46.071166 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:46.067506 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istiod-ca-cert\") pod \"e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:32:46.071166 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:46.067571 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-workload-socket\") pod \"e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:32:46.071166 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:46.068309 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-workload-socket\") pod \"e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:32:46.071166 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:46.020471 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-workload-certs\") pod \"e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:32:46.071166 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:46.068707 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-credential-socket\") pod \"e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:32:46.071166 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:46.020822 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istio-data\") pod \"e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:32:46.132704 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:46.132659 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istiod-ca-cert\") pod \"e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:32:46.288226 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:46.272664 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istio-envoy\") pod \"e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:32:46.288226 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:46.273215 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istio-token\") pod \"e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:32:46.294928 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:46.288946 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69pdb\" (UniqueName: \"kubernetes.io/projected/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-kube-api-access-69pdb\") pod \"e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:32:46.294928 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:46.289247 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istio-podinfo\") pod \"e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:32:46.460928 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:46.460889 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:32:46.713750 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:46.713662 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:32:46.714758 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:46.714736 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:32:46.802949 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:46.802072 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46"] Oct 30 00:32:46.808626 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:46.808589 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:32:46.808795 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:46.808672 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:32:46.808795 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:46.808719 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:32:47.391933 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:32:47.381089 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:32:47.726134 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:47.726049 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" event={"ID":"ff9c1b8a-7343-4566-9611-7dc87cb6fea5","Type":"ContainerStarted","Data":"43b8ad597cf0aaaad4f4de137cc30ef303022bd09c4c21700397e0ae15d15a30"} Oct 30 00:32:47.726134 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:47.726106 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" event={"ID":"ff9c1b8a-7343-4566-9611-7dc87cb6fea5","Type":"ContainerStarted","Data":"c68617f2d0b5d648c767265e6c5becc71c97cad8ec20f9ed89f2012c69d6d52e"} Oct 30 00:32:47.785513 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:47.785451 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" podStartSLOduration=2.785431729 podStartE2EDuration="2.785431729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:47.778521449 +0000 UTC m=+1682.984022633" watchObservedRunningTime="2026-10-30 00:32:47.785431729 +0000 UTC m=+1682.990932910" Oct 30 00:32:48.146306 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.144458 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm"] Oct 30 00:32:48.146306 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.145511 5001 cpu_manager.go:414] "containerMap: removing container" podUID="35bc1599-694f-4f89-9a8a-7667310b9b90" containerName="istio-proxy" Oct 30 00:32:48.146306 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.145534 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35bc1599-694f-4f89-9a8a-7667310b9b90" containerName="istio-proxy" Oct 30 00:32:48.152489 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.150691 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:32:48.176158 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.176127 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-a782e5-openshift-default-dockercfg-vtgkj\"" Oct 30 00:32:48.177111 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.177089 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm"] Oct 30 00:32:48.266676 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.266627 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b41b47fa-39f2-4620-b648-797837a6cd1a-istio-podinfo\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:32:48.266853 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.266694 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-istio-envoy\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:32:48.266853 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.266830 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-credential-socket\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:32:48.267012 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.266966 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g9c6\" (UniqueName: \"kubernetes.io/projected/b41b47fa-39f2-4620-b648-797837a6cd1a-kube-api-access-6g9c6\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:32:48.267012 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.267006 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-istio-data\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:32:48.267164 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.267150 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b41b47fa-39f2-4620-b648-797837a6cd1a-istio-token\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:32:48.267220 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.267207 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-workload-socket\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:32:48.267277 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.267238 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b41b47fa-39f2-4620-b648-797837a6cd1a-istiod-ca-cert\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:32:48.267440 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.267415 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-workload-certs\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:32:48.372999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.372210 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b41b47fa-39f2-4620-b648-797837a6cd1a-istio-podinfo\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:32:48.372999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.372291 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-istio-envoy\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:32:48.372999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.372407 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-credential-socket\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:32:48.385944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.384150 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-credential-socket\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:32:48.385944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.384340 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g9c6\" (UniqueName: \"kubernetes.io/projected/b41b47fa-39f2-4620-b648-797837a6cd1a-kube-api-access-6g9c6\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:32:48.385944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.384399 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-istio-data\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:32:48.385944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.385188 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-istio-data\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:32:48.397981 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.393345 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b41b47fa-39f2-4620-b648-797837a6cd1a-istio-token\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:32:48.397981 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.393442 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-workload-socket\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:32:48.397981 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.393509 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b41b47fa-39f2-4620-b648-797837a6cd1a-istiod-ca-cert\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:32:48.397981 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.393666 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-workload-certs\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:32:48.397981 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.394313 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-workload-socket\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:32:48.397981 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.396097 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b41b47fa-39f2-4620-b648-797837a6cd1a-istiod-ca-cert\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:32:48.418931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.408641 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-workload-certs\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:32:48.432935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.426370 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-istio-envoy\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:32:48.432935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.432270 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g9c6\" (UniqueName: \"kubernetes.io/projected/b41b47fa-39f2-4620-b648-797837a6cd1a-kube-api-access-6g9c6\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:32:48.447326 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.443559 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b41b47fa-39f2-4620-b648-797837a6cd1a-istio-token\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:32:48.464222 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.461295 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:32:48.494641 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.494597 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b41b47fa-39f2-4620-b648-797837a6cd1a-istio-podinfo\") pod \"e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:32:48.505268 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.505226 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:32:48.757732 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.757604 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:32:48.759123 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.759096 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:32:48.777927 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.775124 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:32:48.933394 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.933364 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c1eeb6-7d97d86c79-zmp4b"] Oct 30 00:32:48.938995 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.938969 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c1eeb6-7d97d86c79-zmp4b" Oct 30 00:32:48.942520 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.942491 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e2bb288-7nz5q"] Oct 30 00:32:48.952616 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.952583 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-c1eeb6\"" Oct 30 00:32:48.953475 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.953446 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-c1eeb6-dockercfg-ch698\"" Oct 30 00:32:48.956206 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.956181 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2bb288-7nz5q" Oct 30 00:32:48.963016 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.962988 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c1eeb6-7d97d86c79-zmp4b"] Oct 30 00:32:48.970929 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.970317 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:32:48.970929 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.970378 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:32:48.970929 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.970385 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:32:48.970929 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.970415 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:32:48.971603 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:48.971509 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e2bb288-7nz5q"] Oct 30 00:32:49.094459 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:49.042425 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8e98d30-8619-44a3-81eb-8d2a2e7c2af5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e2bb288-7nz5q\" (UID: \"b8e98d30-8619-44a3-81eb-8d2a2e7c2af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2bb288-7nz5q" Oct 30 00:32:49.094459 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:49.042492 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmf6t\" (UniqueName: \"kubernetes.io/projected/b8e98d30-8619-44a3-81eb-8d2a2e7c2af5-kube-api-access-kmf6t\") pod \"maas-api-revoke-sub-3e2bb288-7nz5q\" (UID: \"b8e98d30-8619-44a3-81eb-8d2a2e7c2af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2bb288-7nz5q" Oct 30 00:32:49.094459 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:49.042541 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks56w\" (UniqueName: \"kubernetes.io/projected/2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c-kube-api-access-ks56w\") pod \"payload-pre-processing-e2e-shared-a-w4-c1eeb6-7d97d86c79-zmp4b\" (UID: \"2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c1eeb6-7d97d86c79-zmp4b" Oct 30 00:32:49.094459 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:49.042679 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-c1eeb6-7d97d86c79-zmp4b\" (UID: \"2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c1eeb6-7d97d86c79-zmp4b" Oct 30 00:32:49.094459 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:49.042792 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8e98d30-8619-44a3-81eb-8d2a2e7c2af5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e2bb288-7nz5q\" (UID: \"b8e98d30-8619-44a3-81eb-8d2a2e7c2af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2bb288-7nz5q" Oct 30 00:32:49.307763 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:49.307716 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:32:49.341672 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:49.313406 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:32:49.385005 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:49.384871 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:32:49.385898 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:49.331661 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8e98d30-8619-44a3-81eb-8d2a2e7c2af5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e2bb288-7nz5q\" (UID: \"b8e98d30-8619-44a3-81eb-8d2a2e7c2af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2bb288-7nz5q" Oct 30 00:32:49.386335 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:49.386314 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8e98d30-8619-44a3-81eb-8d2a2e7c2af5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e2bb288-7nz5q\" (UID: \"b8e98d30-8619-44a3-81eb-8d2a2e7c2af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2bb288-7nz5q" Oct 30 00:32:49.386527 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:49.386510 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmf6t\" (UniqueName: \"kubernetes.io/projected/b8e98d30-8619-44a3-81eb-8d2a2e7c2af5-kube-api-access-kmf6t\") pod \"maas-api-revoke-sub-3e2bb288-7nz5q\" (UID: \"b8e98d30-8619-44a3-81eb-8d2a2e7c2af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2bb288-7nz5q" Oct 30 00:32:49.386670 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:49.386651 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks56w\" (UniqueName: \"kubernetes.io/projected/2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c-kube-api-access-ks56w\") pod \"payload-pre-processing-e2e-shared-a-w4-c1eeb6-7d97d86c79-zmp4b\" (UID: \"2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c1eeb6-7d97d86c79-zmp4b" Oct 30 00:32:49.386888 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:49.386859 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-c1eeb6-7d97d86c79-zmp4b\" (UID: \"2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c1eeb6-7d97d86c79-zmp4b" Oct 30 00:32:49.388027 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:49.388006 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-c1eeb6-7d97d86c79-zmp4b\" (UID: \"2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c1eeb6-7d97d86c79-zmp4b" Oct 30 00:32:49.388210 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:49.339619 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8e98d30-8619-44a3-81eb-8d2a2e7c2af5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e2bb288-7nz5q\" (UID: \"b8e98d30-8619-44a3-81eb-8d2a2e7c2af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2bb288-7nz5q" Oct 30 00:32:49.388893 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:49.388871 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8e98d30-8619-44a3-81eb-8d2a2e7c2af5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e2bb288-7nz5q\" (UID: \"b8e98d30-8619-44a3-81eb-8d2a2e7c2af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2bb288-7nz5q" Oct 30 00:32:49.392703 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:49.392681 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm"] Oct 30 00:32:49.413437 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:49.413406 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmf6t\" (UniqueName: \"kubernetes.io/projected/b8e98d30-8619-44a3-81eb-8d2a2e7c2af5-kube-api-access-kmf6t\") pod \"maas-api-revoke-sub-3e2bb288-7nz5q\" (UID: \"b8e98d30-8619-44a3-81eb-8d2a2e7c2af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2bb288-7nz5q" Oct 30 00:32:49.413967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:49.413893 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks56w\" (UniqueName: \"kubernetes.io/projected/2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c-kube-api-access-ks56w\") pod \"payload-pre-processing-e2e-shared-a-w4-c1eeb6-7d97d86c79-zmp4b\" (UID: \"2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c1eeb6-7d97d86c79-zmp4b" Oct 30 00:32:49.600296 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:49.600261 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2bb288-7nz5q" Oct 30 00:32:49.641127 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:49.641049 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c1eeb6-7d97d86c79-zmp4b" Oct 30 00:32:49.769308 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:49.769262 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" event={"ID":"b41b47fa-39f2-4620-b648-797837a6cd1a","Type":"ContainerStarted","Data":"c18c9fa6e49b4deaa00bf10f962247c1626d4b0e715d446e8281aaabd6755cfe"} Oct 30 00:32:49.770246 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:49.770219 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" event={"ID":"b41b47fa-39f2-4620-b648-797837a6cd1a","Type":"ContainerStarted","Data":"c6b1b6099625ddc906542d0d0f41f21366e3dbd43cb675195ca7f48cb73ab56e"} Oct 30 00:32:49.776071 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:49.776047 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:32:49.811490 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:49.811421 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" podStartSLOduration=1.81140321 podStartE2EDuration="1.81140321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:49.80828739 +0000 UTC m=+1685.013788563" watchObservedRunningTime="2026-10-30 00:32:49.81140321 +0000 UTC m=+1685.016904387" Oct 30 00:32:49.959107 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:49.957158 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e2bb288-7nz5q"] Oct 30 00:32:50.071047 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:50.071018 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c1eeb6-7d97d86c79-zmp4b"] Oct 30 00:32:50.782221 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:50.782190 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:32:50.784111 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:50.784081 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c1eeb6-7d97d86c79-zmp4b" event={"ID":"2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c","Type":"ContainerStarted","Data":"83e85ac636e6cbf91083e1187a5c2965a1a6265f3c27a4ecde281ec1daec8ecb"} Oct 30 00:32:50.784243 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:50.784130 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c1eeb6-7d97d86c79-zmp4b" event={"ID":"2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c","Type":"ContainerStarted","Data":"61e3eb4047d735c4a9d1f18f64ae125c3313dab2cdfbb4a13ced0c8300256adb"} Oct 30 00:32:50.784243 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:50.784176 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c1eeb6-7d97d86c79-zmp4b" Oct 30 00:32:50.786219 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:50.786193 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2bb288-7nz5q" event={"ID":"b8e98d30-8619-44a3-81eb-8d2a2e7c2af5","Type":"ContainerStarted","Data":"67dce500ec036f2aab42964ee8aae4f3307c6b18a03550b56254a6a84256e2cb"} Oct 30 00:32:50.786334 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:50.786237 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2bb288-7nz5q" event={"ID":"b8e98d30-8619-44a3-81eb-8d2a2e7c2af5","Type":"ContainerStarted","Data":"4cd74f81a7a5afe1e6f19102fc841a93bb527fffb690528d01ba056169f1b878"} Oct 30 00:32:50.786693 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:50.786635 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:32:50.787320 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:50.787298 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:32:51.112986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:51.110100 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2bb288-7nz5q" podStartSLOduration=3.110080137 podStartE2EDuration="3.110080137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:50.97715526 +0000 UTC m=+1686.182656468" watchObservedRunningTime="2026-10-30 00:32:51.110080137 +0000 UTC m=+1686.315581318" Oct 30 00:32:51.210063 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:51.209980 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c1eeb6-7d97d86c79-zmp4b" podStartSLOduration=3.209957116 podStartE2EDuration="3.209957116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:51.034377586 +0000 UTC m=+1686.239878778" watchObservedRunningTime="2026-10-30 00:32:51.209957116 +0000 UTC m=+1686.415458312" Oct 30 00:32:51.846155 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:51.846120 5001 generic.go:356] "Generic (PLEG): container finished" podID="b8e98d30-8619-44a3-81eb-8d2a2e7c2af5" containerID="67dce500ec036f2aab42964ee8aae4f3307c6b18a03550b56254a6a84256e2cb" exitCode=0 Oct 30 00:32:51.848029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:51.848000 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2bb288-7nz5q" event={"ID":"b8e98d30-8619-44a3-81eb-8d2a2e7c2af5","Type":"ContainerDied","Data":"67dce500ec036f2aab42964ee8aae4f3307c6b18a03550b56254a6a84256e2cb"} Oct 30 00:32:52.159431 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:52.159320 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p"] Oct 30 00:32:52.180238 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:52.180201 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:32:52.225935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:52.215419 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-63e288-openshift-default-dockercfg-d6zj8\"" Oct 30 00:32:52.236935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:52.234766 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p"] Oct 30 00:32:52.564845 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:52.562412 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-credential-socket\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:32:52.564845 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:52.562486 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-istio-envoy\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:32:52.564845 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:52.562668 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aadaf6fb-2702-4c04-88c9-cbb181049d68-istio-token\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:32:52.564845 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:52.562788 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-workload-certs\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:32:52.564845 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:52.562833 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-workload-socket\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:32:52.564845 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:52.562865 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aadaf6fb-2702-4c04-88c9-cbb181049d68-istio-podinfo\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:32:52.564845 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:52.562961 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kzzp\" (UniqueName: \"kubernetes.io/projected/aadaf6fb-2702-4c04-88c9-cbb181049d68-kube-api-access-7kzzp\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:32:52.564845 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:52.563112 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aadaf6fb-2702-4c04-88c9-cbb181049d68-istiod-ca-cert\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:32:52.564845 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:52.563153 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-istio-data\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:32:52.661013 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:52.660976 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-29f3ca-6765676c44-vvgqp"] Oct 30 00:32:52.743937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:52.728995 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aadaf6fb-2702-4c04-88c9-cbb181049d68-istiod-ca-cert\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:32:52.766701 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:52.766655 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-istio-data\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:32:52.767092 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:52.767068 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-credential-socket\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:32:52.767306 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:52.767274 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-istio-envoy\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:32:52.785408 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:52.785364 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aadaf6fb-2702-4c04-88c9-cbb181049d68-istio-token\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:32:52.785759 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:52.785724 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-workload-certs\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:32:52.785974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:52.785957 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-workload-socket\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:32:52.786114 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:52.786099 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aadaf6fb-2702-4c04-88c9-cbb181049d68-istio-podinfo\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:32:52.786324 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:52.786308 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kzzp\" (UniqueName: \"kubernetes.io/projected/aadaf6fb-2702-4c04-88c9-cbb181049d68-kube-api-access-7kzzp\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:32:52.787777 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:52.787748 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-workload-certs\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:32:52.788289 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:52.788269 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-workload-socket\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:32:52.842976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:52.761006 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aadaf6fb-2702-4c04-88c9-cbb181049d68-istiod-ca-cert\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:32:52.842976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:52.842359 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-istio-data\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:32:52.842976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:52.842870 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-credential-socket\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:32:52.880012 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:52.870002 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-istio-envoy\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:32:52.880012 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:52.870544 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aadaf6fb-2702-4c04-88c9-cbb181049d68-istio-podinfo\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:32:52.981933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:52.969542 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-29f3ca-6765676c44-vvgqp"] Oct 30 00:32:53.024937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:53.024007 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aadaf6fb-2702-4c04-88c9-cbb181049d68-istio-token\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:32:53.028933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:53.025454 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-29f3ca-6765676c44-vvgqp" Oct 30 00:32:53.070451 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:53.070418 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-29f3ca\"" Oct 30 00:32:53.070863 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:53.070840 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-29f3ca-dockercfg-zn9vh\"" Oct 30 00:32:53.125990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:53.113114 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kzzp\" (UniqueName: \"kubernetes.io/projected/aadaf6fb-2702-4c04-88c9-cbb181049d68-kube-api-access-7kzzp\") pod \"e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:32:53.125990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:53.114834 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4c8106e-3826-4f4d-b267-93ecea650c90-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-29f3ca-6765676c44-vvgqp\" (UID: \"d4c8106e-3826-4f4d-b267-93ecea650c90\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-29f3ca-6765676c44-vvgqp" Oct 30 00:32:53.125990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:53.115268 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql6cc\" (UniqueName: \"kubernetes.io/projected/d4c8106e-3826-4f4d-b267-93ecea650c90-kube-api-access-ql6cc\") pod \"payload-processing-e2e-shared-a-w0-29f3ca-6765676c44-vvgqp\" (UID: \"d4c8106e-3826-4f4d-b267-93ecea650c90\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-29f3ca-6765676c44-vvgqp" Oct 30 00:32:53.130326 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:53.130295 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:32:53.239965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:53.238239 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 00:32:53.324621 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:53.324582 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql6cc\" (UniqueName: \"kubernetes.io/projected/d4c8106e-3826-4f4d-b267-93ecea650c90-kube-api-access-ql6cc\") pod \"payload-processing-e2e-shared-a-w0-29f3ca-6765676c44-vvgqp\" (UID: \"d4c8106e-3826-4f4d-b267-93ecea650c90\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-29f3ca-6765676c44-vvgqp" Oct 30 00:32:53.381984 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:53.364337 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4c8106e-3826-4f4d-b267-93ecea650c90-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-29f3ca-6765676c44-vvgqp\" (UID: \"d4c8106e-3826-4f4d-b267-93ecea650c90\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-29f3ca-6765676c44-vvgqp" Oct 30 00:32:53.429184 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:53.367671 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4c8106e-3826-4f4d-b267-93ecea650c90-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-29f3ca-6765676c44-vvgqp\" (UID: \"d4c8106e-3826-4f4d-b267-93ecea650c90\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-29f3ca-6765676c44-vvgqp" Oct 30 00:32:53.448761 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:53.448721 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql6cc\" (UniqueName: \"kubernetes.io/projected/d4c8106e-3826-4f4d-b267-93ecea650c90-kube-api-access-ql6cc\") pod \"payload-processing-e2e-shared-a-w0-29f3ca-6765676c44-vvgqp\" (UID: \"d4c8106e-3826-4f4d-b267-93ecea650c90\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-29f3ca-6765676c44-vvgqp" Oct 30 00:32:53.658014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:53.657214 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-29f3ca-6765676c44-vvgqp" Oct 30 00:32:53.994727 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:53.879000 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:32:54.021880 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:54.021832 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:32:54.069419 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:54.067647 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:32:54.088265 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:54.088014 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:32:54.266168 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:54.266127 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:32:54.488943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:54.322018 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:32:54.488943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:54.469776 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:32:54.518010 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:54.489751 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:32:54.518010 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:54.512566 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:32:54.711074 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:32:54.711037 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaadaf6fb_2702_4c04_88c9_cbb181049d68.slice/crio-3214e1c7056e221cf7887decf7b678da2745a82a7faf17e64a695add10aab935 WatchSource:0}: Error finding container 3214e1c7056e221cf7887decf7b678da2745a82a7faf17e64a695add10aab935: Status 404 returned error can't find the container with id 3214e1c7056e221cf7887decf7b678da2745a82a7faf17e64a695add10aab935 Oct 30 00:32:54.829174 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:54.826154 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:32:54.829174 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:54.827822 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:32:54.829174 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:54.827871 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:32:55.374192 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:55.367609 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:32:55.374192 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:55.367695 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" podUID="4c415517-5484-4905-b9df-8510d1da9ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:32:55.651234 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:55.643308 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p"] Oct 30 00:32:55.776268 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:55.776232 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2bb288-7nz5q" event={"ID":"b8e98d30-8619-44a3-81eb-8d2a2e7c2af5","Type":"ContainerDied","Data":"4cd74f81a7a5afe1e6f19102fc841a93bb527fffb690528d01ba056169f1b878"} Oct 30 00:32:55.777547 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:55.777521 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cd74f81a7a5afe1e6f19102fc841a93bb527fffb690528d01ba056169f1b878" Oct 30 00:32:55.784931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:55.784879 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" event={"ID":"aadaf6fb-2702-4c04-88c9-cbb181049d68","Type":"ContainerStarted","Data":"3214e1c7056e221cf7887decf7b678da2745a82a7faf17e64a695add10aab935"} Oct 30 00:32:55.890153 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:55.890062 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:32:55.906982 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:55.906882 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a782e5-7dddfb8f55-tcmx6"] Oct 30 00:32:55.917180 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:55.917139 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a782e5-7dddfb8f55-tcmx6" Oct 30 00:32:55.937431 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:55.936211 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-a782e5-dockercfg-t6w4g\"" Oct 30 00:32:55.937431 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:55.937141 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-a782e5\"" Oct 30 00:32:55.976123 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:55.976087 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a782e5-7dddfb8f55-tcmx6"] Oct 30 00:32:56.013388 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:55.980933 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2bb288-7nz5q" Oct 30 00:32:56.013388 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.001333 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a782e5-56b6d8bf96-8pss5"] Oct 30 00:32:56.013388 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.003458 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b8e98d30-8619-44a3-81eb-8d2a2e7c2af5" containerName="revoke-keys" Oct 30 00:32:56.013388 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.003478 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8e98d30-8619-44a3-81eb-8d2a2e7c2af5" containerName="revoke-keys" Oct 30 00:32:56.013388 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.003666 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8e98d30-8619-44a3-81eb-8d2a2e7c2af5" containerName="revoke-keys" Oct 30 00:32:56.035088 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.032440 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a782e5-56b6d8bf96-8pss5"] Oct 30 00:32:56.035088 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.032573 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a782e5-56b6d8bf96-8pss5" Oct 30 00:32:56.043987 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.043960 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fad7d195-9gjm9"] Oct 30 00:32:56.100546 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.082097 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8e98d30-8619-44a3-81eb-8d2a2e7c2af5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8e98d30-8619-44a3-81eb-8d2a2e7c2af5-maas-api-xks-ca\") pod \"b8e98d30-8619-44a3-81eb-8d2a2e7c2af5\" (UID: \"b8e98d30-8619-44a3-81eb-8d2a2e7c2af5\") " Oct 30 00:32:56.100546 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.082392 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8e98d30-8619-44a3-81eb-8d2a2e7c2af5-kube-api-access-kmf6t\" (UniqueName: \"kubernetes.io/projected/b8e98d30-8619-44a3-81eb-8d2a2e7c2af5-kube-api-access-kmf6t\") pod \"b8e98d30-8619-44a3-81eb-8d2a2e7c2af5\" (UID: \"b8e98d30-8619-44a3-81eb-8d2a2e7c2af5\") " Oct 30 00:32:56.100546 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.082458 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8e98d30-8619-44a3-81eb-8d2a2e7c2af5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8e98d30-8619-44a3-81eb-8d2a2e7c2af5-maas-api-openshift-ca\") pod \"b8e98d30-8619-44a3-81eb-8d2a2e7c2af5\" (UID: \"b8e98d30-8619-44a3-81eb-8d2a2e7c2af5\") " Oct 30 00:32:56.100546 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.082987 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8e98d30-8619-44a3-81eb-8d2a2e7c2af5-maas-api-openshift-ca" pod "b8e98d30-8619-44a3-81eb-8d2a2e7c2af5" (UID: "b8e98d30-8619-44a3-81eb-8d2a2e7c2af5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:32:56.100546 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.089311 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swtsf\" (UniqueName: \"kubernetes.io/projected/3a97385f-5ca1-41db-9a2a-66cbc70baad2-kube-api-access-swtsf\") pod \"payload-pre-processing-e2e-shared-a-w2-a782e5-7dddfb8f55-tcmx6\" (UID: \"3a97385f-5ca1-41db-9a2a-66cbc70baad2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a782e5-7dddfb8f55-tcmx6" Oct 30 00:32:56.100546 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.089506 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a97385f-5ca1-41db-9a2a-66cbc70baad2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-a782e5-7dddfb8f55-tcmx6\" (UID: \"3a97385f-5ca1-41db-9a2a-66cbc70baad2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a782e5-7dddfb8f55-tcmx6" Oct 30 00:32:56.100546 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.089630 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8e98d30-8619-44a3-81eb-8d2a2e7c2af5-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:56.100546 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.095122 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8e98d30-8619-44a3-81eb-8d2a2e7c2af5-kube-api-access-kmf6t" pod "b8e98d30-8619-44a3-81eb-8d2a2e7c2af5" (UID: "b8e98d30-8619-44a3-81eb-8d2a2e7c2af5"). InnerVolumeSpecName "kube-api-access-kmf6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:32:56.100546 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.099831 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fad7d195-9gjm9"] Oct 30 00:32:56.100546 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.099889 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-29f3ca-6765676c44-vvgqp"] Oct 30 00:32:56.100546 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.099984 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad7d195-9gjm9" Oct 30 00:32:56.103431 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.103386 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8e98d30-8619-44a3-81eb-8d2a2e7c2af5-maas-api-xks-ca" pod "b8e98d30-8619-44a3-81eb-8d2a2e7c2af5" (UID: "b8e98d30-8619-44a3-81eb-8d2a2e7c2af5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:32:56.195124 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.195069 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w8gw\" (UniqueName: \"kubernetes.io/projected/d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea-kube-api-access-4w8gw\") pod \"payload-processing-e2e-shared-a-w2-a782e5-56b6d8bf96-8pss5\" (UID: \"d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a782e5-56b6d8bf96-8pss5" Oct 30 00:32:56.195295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.195163 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca471334-1301-43ac-886e-90d3610b5ea9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fad7d195-9gjm9\" (UID: \"ca471334-1301-43ac-886e-90d3610b5ea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad7d195-9gjm9" Oct 30 00:32:56.195295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.195276 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swtsf\" (UniqueName: \"kubernetes.io/projected/3a97385f-5ca1-41db-9a2a-66cbc70baad2-kube-api-access-swtsf\") pod \"payload-pre-processing-e2e-shared-a-w2-a782e5-7dddfb8f55-tcmx6\" (UID: \"3a97385f-5ca1-41db-9a2a-66cbc70baad2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a782e5-7dddfb8f55-tcmx6" Oct 30 00:32:56.195465 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.195446 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a97385f-5ca1-41db-9a2a-66cbc70baad2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-a782e5-7dddfb8f55-tcmx6\" (UID: \"3a97385f-5ca1-41db-9a2a-66cbc70baad2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a782e5-7dddfb8f55-tcmx6" Oct 30 00:32:56.195610 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.195593 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x699q\" (UniqueName: \"kubernetes.io/projected/ca471334-1301-43ac-886e-90d3610b5ea9-kube-api-access-x699q\") pod \"maas-api-revoke-sub-fad7d195-9gjm9\" (UID: \"ca471334-1301-43ac-886e-90d3610b5ea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad7d195-9gjm9" Oct 30 00:32:56.195727 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.195631 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca471334-1301-43ac-886e-90d3610b5ea9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fad7d195-9gjm9\" (UID: \"ca471334-1301-43ac-886e-90d3610b5ea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad7d195-9gjm9" Oct 30 00:32:56.195727 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.195678 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-a782e5-56b6d8bf96-8pss5\" (UID: \"d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a782e5-56b6d8bf96-8pss5" Oct 30 00:32:56.196331 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.196282 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8e98d30-8619-44a3-81eb-8d2a2e7c2af5-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:56.196465 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.196335 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmf6t\" (UniqueName: \"kubernetes.io/projected/b8e98d30-8619-44a3-81eb-8d2a2e7c2af5-kube-api-access-kmf6t\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:32:56.213932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.206501 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a97385f-5ca1-41db-9a2a-66cbc70baad2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-a782e5-7dddfb8f55-tcmx6\" (UID: \"3a97385f-5ca1-41db-9a2a-66cbc70baad2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a782e5-7dddfb8f55-tcmx6" Oct 30 00:32:56.237500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.237466 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swtsf\" (UniqueName: \"kubernetes.io/projected/3a97385f-5ca1-41db-9a2a-66cbc70baad2-kube-api-access-swtsf\") pod \"payload-pre-processing-e2e-shared-a-w2-a782e5-7dddfb8f55-tcmx6\" (UID: \"3a97385f-5ca1-41db-9a2a-66cbc70baad2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a782e5-7dddfb8f55-tcmx6" Oct 30 00:32:56.298113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.298071 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x699q\" (UniqueName: \"kubernetes.io/projected/ca471334-1301-43ac-886e-90d3610b5ea9-kube-api-access-x699q\") pod \"maas-api-revoke-sub-fad7d195-9gjm9\" (UID: \"ca471334-1301-43ac-886e-90d3610b5ea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad7d195-9gjm9" Oct 30 00:32:56.298264 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.298127 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca471334-1301-43ac-886e-90d3610b5ea9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fad7d195-9gjm9\" (UID: \"ca471334-1301-43ac-886e-90d3610b5ea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad7d195-9gjm9" Oct 30 00:32:56.298264 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.298166 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-a782e5-56b6d8bf96-8pss5\" (UID: \"d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a782e5-56b6d8bf96-8pss5" Oct 30 00:32:56.298264 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.298246 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w8gw\" (UniqueName: \"kubernetes.io/projected/d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea-kube-api-access-4w8gw\") pod \"payload-processing-e2e-shared-a-w2-a782e5-56b6d8bf96-8pss5\" (UID: \"d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a782e5-56b6d8bf96-8pss5" Oct 30 00:32:56.298813 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.298305 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca471334-1301-43ac-886e-90d3610b5ea9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fad7d195-9gjm9\" (UID: \"ca471334-1301-43ac-886e-90d3610b5ea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad7d195-9gjm9" Oct 30 00:32:56.298980 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.298952 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca471334-1301-43ac-886e-90d3610b5ea9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fad7d195-9gjm9\" (UID: \"ca471334-1301-43ac-886e-90d3610b5ea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad7d195-9gjm9" Oct 30 00:32:56.299168 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.299143 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-a782e5-56b6d8bf96-8pss5\" (UID: \"d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a782e5-56b6d8bf96-8pss5" Oct 30 00:32:56.301077 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.301054 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca471334-1301-43ac-886e-90d3610b5ea9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fad7d195-9gjm9\" (UID: \"ca471334-1301-43ac-886e-90d3610b5ea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad7d195-9gjm9" Oct 30 00:32:56.303447 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.303426 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a782e5-7dddfb8f55-tcmx6" Oct 30 00:32:56.318685 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.318666 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w8gw\" (UniqueName: \"kubernetes.io/projected/d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea-kube-api-access-4w8gw\") pod \"payload-processing-e2e-shared-a-w2-a782e5-56b6d8bf96-8pss5\" (UID: \"d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a782e5-56b6d8bf96-8pss5" Oct 30 00:32:56.319120 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.319099 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x699q\" (UniqueName: \"kubernetes.io/projected/ca471334-1301-43ac-886e-90d3610b5ea9-kube-api-access-x699q\") pod \"maas-api-revoke-sub-fad7d195-9gjm9\" (UID: \"ca471334-1301-43ac-886e-90d3610b5ea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad7d195-9gjm9" Oct 30 00:32:56.377434 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.377156 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a782e5-56b6d8bf96-8pss5" Oct 30 00:32:56.557864 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.557832 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad7d195-9gjm9" Oct 30 00:32:56.621404 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.621321 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a782e5-7dddfb8f55-tcmx6"] Oct 30 00:32:56.629858 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:32:56.629826 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a97385f_5ca1_41db_9a2a_66cbc70baad2.slice/crio-e9d8a19cb82fb79d426f4528448c9af95d9d645b7394fb8e390c3ddc231866d8 WatchSource:0}: Error finding container e9d8a19cb82fb79d426f4528448c9af95d9d645b7394fb8e390c3ddc231866d8: Status 404 returned error can't find the container with id e9d8a19cb82fb79d426f4528448c9af95d9d645b7394fb8e390c3ddc231866d8 Oct 30 00:32:56.692708 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:32:56.692676 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3b705e2_ecf3_4a35_a8e1_6111fd40a4ea.slice/crio-eac22097e50d9b351e35146b5707bb25da967ad3678dcf82aed32bd32d6d1350 WatchSource:0}: Error finding container eac22097e50d9b351e35146b5707bb25da967ad3678dcf82aed32bd32d6d1350: Status 404 returned error can't find the container with id eac22097e50d9b351e35146b5707bb25da967ad3678dcf82aed32bd32d6d1350 Oct 30 00:32:56.693329 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.693292 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a782e5-56b6d8bf96-8pss5"] Oct 30 00:32:56.794944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.794880 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a782e5-7dddfb8f55-tcmx6" event={"ID":"3a97385f-5ca1-41db-9a2a-66cbc70baad2","Type":"ContainerStarted","Data":"4edf7887e39c5427a700c76a4d8230032c53349f5532230dd1762d3e302bd451"} Oct 30 00:32:56.795089 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.794973 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a782e5-7dddfb8f55-tcmx6" event={"ID":"3a97385f-5ca1-41db-9a2a-66cbc70baad2","Type":"ContainerStarted","Data":"e9d8a19cb82fb79d426f4528448c9af95d9d645b7394fb8e390c3ddc231866d8"} Oct 30 00:32:56.795168 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.795131 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a782e5-7dddfb8f55-tcmx6" Oct 30 00:32:56.796881 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.796854 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" event={"ID":"aadaf6fb-2702-4c04-88c9-cbb181049d68","Type":"ContainerStarted","Data":"35460dd73d82bac9413321658766b2cade487258b6d5c905472184b20bbed8d6"} Oct 30 00:32:56.798621 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.798595 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a782e5-56b6d8bf96-8pss5" event={"ID":"d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea","Type":"ContainerStarted","Data":"a687cb927a21b9784bf2a4f2d63fbd9a1ca9aea48c31f4791fca9ac3041fa850"} Oct 30 00:32:56.798752 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.798644 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a782e5-56b6d8bf96-8pss5" event={"ID":"d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea","Type":"ContainerStarted","Data":"eac22097e50d9b351e35146b5707bb25da967ad3678dcf82aed32bd32d6d1350"} Oct 30 00:32:56.798752 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.798685 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a782e5-56b6d8bf96-8pss5" Oct 30 00:32:56.800394 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.800368 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-29f3ca-6765676c44-vvgqp" event={"ID":"d4c8106e-3826-4f4d-b267-93ecea650c90","Type":"ContainerStarted","Data":"5145862a61b8d6ac9966ff9c40475f0ae6d733ccdc3404f250ce8b1617fe5821"} Oct 30 00:32:56.800530 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.800413 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-29f3ca-6765676c44-vvgqp" event={"ID":"d4c8106e-3826-4f4d-b267-93ecea650c90","Type":"ContainerStarted","Data":"49e61459e072d7664874614527fd35dd32d08d1765b06f8362bb987ee43e3655"} Oct 30 00:32:56.800530 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.800457 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-29f3ca-6765676c44-vvgqp" Oct 30 00:32:56.800530 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.800472 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2bb288-7nz5q" Oct 30 00:32:56.831020 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.830979 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a782e5-7dddfb8f55-tcmx6" podStartSLOduration=2.83096681 podStartE2EDuration="2.83096681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:56.829367299 +0000 UTC m=+1692.034868498" watchObservedRunningTime="2026-10-30 00:32:56.83096681 +0000 UTC m=+1692.036467992" Oct 30 00:32:56.868261 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:32:56.868232 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca471334_1301_43ac_886e_90d3610b5ea9.slice/crio-9516e19bc0c906b28d3c3aa5c73c8c8f770d51a67d3acbbf4361fdc957d6c82e WatchSource:0}: Error finding container 9516e19bc0c906b28d3c3aa5c73c8c8f770d51a67d3acbbf4361fdc957d6c82e: Status 404 returned error can't find the container with id 9516e19bc0c906b28d3c3aa5c73c8c8f770d51a67d3acbbf4361fdc957d6c82e Oct 30 00:32:56.870182 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.869733 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fad7d195-9gjm9"] Oct 30 00:32:56.874879 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.874824 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a782e5-56b6d8bf96-8pss5" podStartSLOduration=2.874805006 podStartE2EDuration="2.874805006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:56.866247875 +0000 UTC m=+1692.071749058" watchObservedRunningTime="2026-10-30 00:32:56.874805006 +0000 UTC m=+1692.080306189" Oct 30 00:32:56.910499 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.910450 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" podStartSLOduration=4.910431826 podStartE2EDuration="4.910431826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:56.906072942 +0000 UTC m=+1692.111574124" watchObservedRunningTime="2026-10-30 00:32:56.910431826 +0000 UTC m=+1692.115933008" Oct 30 00:32:56.957291 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:56.957200 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-29f3ca-6765676c44-vvgqp" podStartSLOduration=4.957186297 podStartE2EDuration="4.957186297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:56.952982593 +0000 UTC m=+1692.158483771" watchObservedRunningTime="2026-10-30 00:32:56.957186297 +0000 UTC m=+1692.162687511" Oct 30 00:32:57.139163 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:57.139132 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:32:57.152248 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:57.152068 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:32:57.812759 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:57.812725 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad7d195-9gjm9" event={"ID":"ca471334-1301-43ac-886e-90d3610b5ea9","Type":"ContainerStarted","Data":"14fa5ee31c34fa120e28c8f1b1029476ec09ade105099cac16c2a6192be1bc6b"} Oct 30 00:32:57.813284 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:57.812777 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad7d195-9gjm9" event={"ID":"ca471334-1301-43ac-886e-90d3610b5ea9","Type":"ContainerStarted","Data":"9516e19bc0c906b28d3c3aa5c73c8c8f770d51a67d3acbbf4361fdc957d6c82e"} Oct 30 00:32:57.813511 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:57.813489 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:32:57.813980 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:57.813959 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:32:57.852873 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:57.852824 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad7d195-9gjm9" podStartSLOduration=2.8528099080000002 podStartE2EDuration="2.852809908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:57.848672766 +0000 UTC m=+1693.054173948" watchObservedRunningTime="2026-10-30 00:32:57.852809908 +0000 UTC m=+1693.058311089" Oct 30 00:32:58.830689 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:58.830642 5001 generic.go:356] "Generic (PLEG): container finished" podID="ca471334-1301-43ac-886e-90d3610b5ea9" containerID="14fa5ee31c34fa120e28c8f1b1029476ec09ade105099cac16c2a6192be1bc6b" exitCode=0 Oct 30 00:32:58.831282 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:32:58.830749 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad7d195-9gjm9" event={"ID":"ca471334-1301-43ac-886e-90d3610b5ea9","Type":"ContainerDied","Data":"14fa5ee31c34fa120e28c8f1b1029476ec09ade105099cac16c2a6192be1bc6b"} Oct 30 00:33:00.018292 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:00.018264 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad7d195-9gjm9" Oct 30 00:33:00.058681 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:00.058363 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca471334-1301-43ac-886e-90d3610b5ea9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca471334-1301-43ac-886e-90d3610b5ea9-maas-api-openshift-ca\") pod \"ca471334-1301-43ac-886e-90d3610b5ea9\" (UID: \"ca471334-1301-43ac-886e-90d3610b5ea9\") " Oct 30 00:33:00.058886 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:00.058849 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca471334-1301-43ac-886e-90d3610b5ea9-maas-api-openshift-ca" pod "ca471334-1301-43ac-886e-90d3610b5ea9" (UID: "ca471334-1301-43ac-886e-90d3610b5ea9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:33:00.059038 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:00.059014 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca471334-1301-43ac-886e-90d3610b5ea9-kube-api-access-x699q\" (UniqueName: \"kubernetes.io/projected/ca471334-1301-43ac-886e-90d3610b5ea9-kube-api-access-x699q\") pod \"ca471334-1301-43ac-886e-90d3610b5ea9\" (UID: \"ca471334-1301-43ac-886e-90d3610b5ea9\") " Oct 30 00:33:00.059161 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:00.059050 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca471334-1301-43ac-886e-90d3610b5ea9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca471334-1301-43ac-886e-90d3610b5ea9-maas-api-xks-ca\") pod \"ca471334-1301-43ac-886e-90d3610b5ea9\" (UID: \"ca471334-1301-43ac-886e-90d3610b5ea9\") " Oct 30 00:33:00.060216 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:00.060193 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca471334-1301-43ac-886e-90d3610b5ea9-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:33:00.061496 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:00.061466 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca471334-1301-43ac-886e-90d3610b5ea9-kube-api-access-x699q" pod "ca471334-1301-43ac-886e-90d3610b5ea9" (UID: "ca471334-1301-43ac-886e-90d3610b5ea9"). InnerVolumeSpecName "kube-api-access-x699q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:33:00.061652 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:00.061631 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca471334-1301-43ac-886e-90d3610b5ea9-maas-api-xks-ca" pod "ca471334-1301-43ac-886e-90d3610b5ea9" (UID: "ca471334-1301-43ac-886e-90d3610b5ea9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:33:00.161234 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:00.161154 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x699q\" (UniqueName: \"kubernetes.io/projected/ca471334-1301-43ac-886e-90d3610b5ea9-kube-api-access-x699q\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:33:00.161234 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:00.161182 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca471334-1301-43ac-886e-90d3610b5ea9-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:33:00.852273 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:00.852246 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad7d195-9gjm9" Oct 30 00:33:00.852555 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:00.852233 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad7d195-9gjm9" event={"ID":"ca471334-1301-43ac-886e-90d3610b5ea9","Type":"ContainerDied","Data":"9516e19bc0c906b28d3c3aa5c73c8c8f770d51a67d3acbbf4361fdc957d6c82e"} Oct 30 00:33:00.852555 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:00.852398 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9516e19bc0c906b28d3c3aa5c73c8c8f770d51a67d3acbbf4361fdc957d6c82e" Oct 30 00:33:01.317641 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:01.317608 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3e94c5f-hr2vp"] Oct 30 00:33:01.318304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:01.318289 5001 cpu_manager.go:414] "containerMap: removing container" podUID="ca471334-1301-43ac-886e-90d3610b5ea9" containerName="revoke-keys" Oct 30 00:33:01.318351 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:01.318304 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca471334-1301-43ac-886e-90d3610b5ea9" containerName="revoke-keys" Oct 30 00:33:01.318413 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:01.318404 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca471334-1301-43ac-886e-90d3610b5ea9" containerName="revoke-keys" Oct 30 00:33:01.322180 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:01.322161 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e94c5f-hr2vp" Oct 30 00:33:01.336322 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:01.336302 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:33:01.346391 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:01.346367 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:33:01.346520 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:01.346471 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:33:01.346600 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:01.346523 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:33:01.352365 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:01.352338 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3e94c5f-hr2vp"] Oct 30 00:33:01.384140 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:01.384110 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s2ws\" (UniqueName: \"kubernetes.io/projected/22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417-kube-api-access-9s2ws\") pod \"maas-api-revoke-sub-f3e94c5f-hr2vp\" (UID: \"22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e94c5f-hr2vp" Oct 30 00:33:01.384254 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:01.384181 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3e94c5f-hr2vp\" (UID: \"22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e94c5f-hr2vp" Oct 30 00:33:01.384373 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:01.384346 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3e94c5f-hr2vp\" (UID: \"22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e94c5f-hr2vp" Oct 30 00:33:01.485810 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:01.485772 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3e94c5f-hr2vp\" (UID: \"22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e94c5f-hr2vp" Oct 30 00:33:01.486093 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:01.486069 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s2ws\" (UniqueName: \"kubernetes.io/projected/22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417-kube-api-access-9s2ws\") pod \"maas-api-revoke-sub-f3e94c5f-hr2vp\" (UID: \"22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e94c5f-hr2vp" Oct 30 00:33:01.486163 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:01.486147 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3e94c5f-hr2vp\" (UID: \"22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e94c5f-hr2vp" Oct 30 00:33:01.487531 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:01.487503 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3e94c5f-hr2vp\" (UID: \"22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e94c5f-hr2vp" Oct 30 00:33:01.488854 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:01.488831 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3e94c5f-hr2vp\" (UID: \"22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e94c5f-hr2vp" Oct 30 00:33:01.507193 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:01.507166 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s2ws\" (UniqueName: \"kubernetes.io/projected/22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417-kube-api-access-9s2ws\") pod \"maas-api-revoke-sub-f3e94c5f-hr2vp\" (UID: \"22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e94c5f-hr2vp" Oct 30 00:33:01.636023 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:01.635942 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e94c5f-hr2vp" Oct 30 00:33:02.092525 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:02.092153 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c1eeb6-7d97d86c79-zmp4b" Oct 30 00:33:02.183292 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:33:02.183256 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22ba3bbe_857d_4ec4_8ca7_2e7cfcbc1417.slice/crio-00ec848039d4275814ef116849ee50affd72a5e0d7c79c395ec83fd7a47a1b51 WatchSource:0}: Error finding container 00ec848039d4275814ef116849ee50affd72a5e0d7c79c395ec83fd7a47a1b51: Status 404 returned error can't find the container with id 00ec848039d4275814ef116849ee50affd72a5e0d7c79c395ec83fd7a47a1b51 Oct 30 00:33:02.185868 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:02.185827 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3e94c5f-hr2vp"] Oct 30 00:33:03.265429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:03.262900 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e94c5f-hr2vp" event={"ID":"22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417","Type":"ContainerStarted","Data":"56b2bfcf34ff59af50544c34c8e0f8981dc208cbb20cf97436220bcc373df1b1"} Oct 30 00:33:03.265429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:03.263069 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e94c5f-hr2vp" event={"ID":"22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417","Type":"ContainerStarted","Data":"00ec848039d4275814ef116849ee50affd72a5e0d7c79c395ec83fd7a47a1b51"} Oct 30 00:33:03.307016 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:03.306369 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e94c5f-hr2vp" podStartSLOduration=2.30635143 podStartE2EDuration="2.30635143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:03.300040296 +0000 UTC m=+1698.505541503" watchObservedRunningTime="2026-10-30 00:33:03.30635143 +0000 UTC m=+1698.511852612" Oct 30 00:33:04.272970 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:04.272936 5001 generic.go:356] "Generic (PLEG): container finished" podID="22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417" containerID="56b2bfcf34ff59af50544c34c8e0f8981dc208cbb20cf97436220bcc373df1b1" exitCode=0 Oct 30 00:33:04.273455 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:04.273034 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e94c5f-hr2vp" event={"ID":"22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417","Type":"ContainerDied","Data":"56b2bfcf34ff59af50544c34c8e0f8981dc208cbb20cf97436220bcc373df1b1"} Oct 30 00:33:05.505417 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:05.505392 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e94c5f-hr2vp" Oct 30 00:33:05.621213 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:05.621136 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417-maas-api-openshift-ca\") pod \"22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417\" (UID: \"22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417\") " Oct 30 00:33:05.621388 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:05.621265 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417-kube-api-access-9s2ws\" (UniqueName: \"kubernetes.io/projected/22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417-kube-api-access-9s2ws\") pod \"22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417\" (UID: \"22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417\") " Oct 30 00:33:05.621470 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:05.621399 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417-maas-api-xks-ca\") pod \"22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417\" (UID: \"22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417\") " Oct 30 00:33:05.621533 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:05.621489 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417-maas-api-openshift-ca" pod "22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417" (UID: "22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:33:05.622008 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:05.621989 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:33:05.623633 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:05.623606 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417-maas-api-xks-ca" pod "22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417" (UID: "22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:33:05.623736 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:05.623664 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417-kube-api-access-9s2ws" pod "22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417" (UID: "22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417"). InnerVolumeSpecName "kube-api-access-9s2ws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:33:05.724048 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:05.723990 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:33:05.724320 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:05.724288 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s2ws\" (UniqueName: \"kubernetes.io/projected/22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417-kube-api-access-9s2ws\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:33:05.951987 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:05.951877 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e45580f4-lwvdl"] Oct 30 00:33:05.952890 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:05.952868 5001 cpu_manager.go:414] "containerMap: removing container" podUID="22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417" containerName="revoke-keys" Oct 30 00:33:05.953030 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:05.952895 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417" containerName="revoke-keys" Oct 30 00:33:05.953173 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:05.953159 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417" containerName="revoke-keys" Oct 30 00:33:05.957462 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:05.957437 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e45580f4-lwvdl" Oct 30 00:33:06.012667 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:06.012581 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e45580f4-lwvdl"] Oct 30 00:33:06.176835 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:06.167110 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df407c8d-1211-43cc-be72-7096f158be35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e45580f4-lwvdl\" (UID: \"df407c8d-1211-43cc-be72-7096f158be35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e45580f4-lwvdl" Oct 30 00:33:06.176835 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:06.167382 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btcc2\" (UniqueName: \"kubernetes.io/projected/df407c8d-1211-43cc-be72-7096f158be35-kube-api-access-btcc2\") pod \"maas-api-revoke-sub-e45580f4-lwvdl\" (UID: \"df407c8d-1211-43cc-be72-7096f158be35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e45580f4-lwvdl" Oct 30 00:33:06.176835 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:06.167480 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df407c8d-1211-43cc-be72-7096f158be35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e45580f4-lwvdl\" (UID: \"df407c8d-1211-43cc-be72-7096f158be35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e45580f4-lwvdl" Oct 30 00:33:06.269360 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:06.269327 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btcc2\" (UniqueName: \"kubernetes.io/projected/df407c8d-1211-43cc-be72-7096f158be35-kube-api-access-btcc2\") pod \"maas-api-revoke-sub-e45580f4-lwvdl\" (UID: \"df407c8d-1211-43cc-be72-7096f158be35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e45580f4-lwvdl" Oct 30 00:33:06.269567 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:06.269403 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df407c8d-1211-43cc-be72-7096f158be35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e45580f4-lwvdl\" (UID: \"df407c8d-1211-43cc-be72-7096f158be35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e45580f4-lwvdl" Oct 30 00:33:06.269567 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:06.269464 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df407c8d-1211-43cc-be72-7096f158be35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e45580f4-lwvdl\" (UID: \"df407c8d-1211-43cc-be72-7096f158be35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e45580f4-lwvdl" Oct 30 00:33:06.270243 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:06.270218 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df407c8d-1211-43cc-be72-7096f158be35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e45580f4-lwvdl\" (UID: \"df407c8d-1211-43cc-be72-7096f158be35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e45580f4-lwvdl" Oct 30 00:33:06.271950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:06.271905 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df407c8d-1211-43cc-be72-7096f158be35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e45580f4-lwvdl\" (UID: \"df407c8d-1211-43cc-be72-7096f158be35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e45580f4-lwvdl" Oct 30 00:33:06.307621 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:06.307588 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btcc2\" (UniqueName: \"kubernetes.io/projected/df407c8d-1211-43cc-be72-7096f158be35-kube-api-access-btcc2\") pod \"maas-api-revoke-sub-e45580f4-lwvdl\" (UID: \"df407c8d-1211-43cc-be72-7096f158be35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e45580f4-lwvdl" Oct 30 00:33:06.311133 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:06.311109 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e45580f4-lwvdl" Oct 30 00:33:06.316542 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:06.316518 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e94c5f-hr2vp" event={"ID":"22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417","Type":"ContainerDied","Data":"00ec848039d4275814ef116849ee50affd72a5e0d7c79c395ec83fd7a47a1b51"} Oct 30 00:33:06.316634 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:06.316566 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00ec848039d4275814ef116849ee50affd72a5e0d7c79c395ec83fd7a47a1b51" Oct 30 00:33:06.316634 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:06.316531 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e94c5f-hr2vp" Oct 30 00:33:06.662417 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:06.662385 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e45580f4-lwvdl"] Oct 30 00:33:07.327535 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:07.327500 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e45580f4-lwvdl" event={"ID":"df407c8d-1211-43cc-be72-7096f158be35","Type":"ContainerStarted","Data":"b4e288c518f6f4080239560e51975cd36b65a01d961d3d936ad83e00a9ef5b0f"} Oct 30 00:33:07.327718 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:07.327554 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e45580f4-lwvdl" event={"ID":"df407c8d-1211-43cc-be72-7096f158be35","Type":"ContainerStarted","Data":"de13abdff4fe5ae880e39abaf8c2c7189d41abf8b564e2fc546c49ea4854c33f"} Oct 30 00:33:07.367098 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:07.367039 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e45580f4-lwvdl" podStartSLOduration=2.367026213 podStartE2EDuration="2.367026213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:07.361690263 +0000 UTC m=+1702.567191446" watchObservedRunningTime="2026-10-30 00:33:07.367026213 +0000 UTC m=+1702.572527393" Oct 30 00:33:07.816484 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:07.816455 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a782e5-7dddfb8f55-tcmx6" Oct 30 00:33:07.816977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:07.816520 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a782e5-56b6d8bf96-8pss5" Oct 30 00:33:07.816977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:07.816555 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-29f3ca-6765676c44-vvgqp" Oct 30 00:33:08.342482 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:08.342443 5001 generic.go:356] "Generic (PLEG): container finished" podID="df407c8d-1211-43cc-be72-7096f158be35" containerID="b4e288c518f6f4080239560e51975cd36b65a01d961d3d936ad83e00a9ef5b0f" exitCode=0 Oct 30 00:33:08.342673 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:08.342485 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e45580f4-lwvdl" event={"ID":"df407c8d-1211-43cc-be72-7096f158be35","Type":"ContainerDied","Data":"b4e288c518f6f4080239560e51975cd36b65a01d961d3d936ad83e00a9ef5b0f"} Oct 30 00:33:09.517982 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:09.517950 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e45580f4-lwvdl" Oct 30 00:33:09.626853 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:09.626762 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df407c8d-1211-43cc-be72-7096f158be35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df407c8d-1211-43cc-be72-7096f158be35-maas-api-openshift-ca\") pod \"df407c8d-1211-43cc-be72-7096f158be35\" (UID: \"df407c8d-1211-43cc-be72-7096f158be35\") " Oct 30 00:33:09.626853 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:09.626814 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df407c8d-1211-43cc-be72-7096f158be35-kube-api-access-btcc2\" (UniqueName: \"kubernetes.io/projected/df407c8d-1211-43cc-be72-7096f158be35-kube-api-access-btcc2\") pod \"df407c8d-1211-43cc-be72-7096f158be35\" (UID: \"df407c8d-1211-43cc-be72-7096f158be35\") " Oct 30 00:33:09.627104 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:09.626860 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df407c8d-1211-43cc-be72-7096f158be35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df407c8d-1211-43cc-be72-7096f158be35-maas-api-xks-ca\") pod \"df407c8d-1211-43cc-be72-7096f158be35\" (UID: \"df407c8d-1211-43cc-be72-7096f158be35\") " Oct 30 00:33:09.627187 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:09.627161 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df407c8d-1211-43cc-be72-7096f158be35-maas-api-openshift-ca" pod "df407c8d-1211-43cc-be72-7096f158be35" (UID: "df407c8d-1211-43cc-be72-7096f158be35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:33:09.627610 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:09.627588 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df407c8d-1211-43cc-be72-7096f158be35-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:33:09.629032 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:09.629008 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df407c8d-1211-43cc-be72-7096f158be35-maas-api-xks-ca" pod "df407c8d-1211-43cc-be72-7096f158be35" (UID: "df407c8d-1211-43cc-be72-7096f158be35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:33:09.629032 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:09.629021 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df407c8d-1211-43cc-be72-7096f158be35-kube-api-access-btcc2" pod "df407c8d-1211-43cc-be72-7096f158be35" (UID: "df407c8d-1211-43cc-be72-7096f158be35"). InnerVolumeSpecName "kube-api-access-btcc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:33:09.729347 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:09.729312 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btcc2\" (UniqueName: \"kubernetes.io/projected/df407c8d-1211-43cc-be72-7096f158be35-kube-api-access-btcc2\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:33:09.729347 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:09.729342 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df407c8d-1211-43cc-be72-7096f158be35-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:33:10.115128 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:10.115090 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c061895-7m288"] Oct 30 00:33:10.115951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:10.115932 5001 cpu_manager.go:414] "containerMap: removing container" podUID="df407c8d-1211-43cc-be72-7096f158be35" containerName="revoke-keys" Oct 30 00:33:10.116006 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:10.115952 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df407c8d-1211-43cc-be72-7096f158be35" containerName="revoke-keys" Oct 30 00:33:10.116215 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:10.116203 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="df407c8d-1211-43cc-be72-7096f158be35" containerName="revoke-keys" Oct 30 00:33:10.120074 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:10.120056 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c061895-7m288" Oct 30 00:33:10.143535 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:10.141637 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf8f0e8a-689f-42cd-9e12-106da73dd5b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c061895-7m288\" (UID: \"bf8f0e8a-689f-42cd-9e12-106da73dd5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c061895-7m288" Oct 30 00:33:10.143535 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:10.141784 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46pfd\" (UniqueName: \"kubernetes.io/projected/bf8f0e8a-689f-42cd-9e12-106da73dd5b1-kube-api-access-46pfd\") pod \"maas-api-revoke-sub-5c061895-7m288\" (UID: \"bf8f0e8a-689f-42cd-9e12-106da73dd5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c061895-7m288" Oct 30 00:33:10.143535 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:10.141938 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf8f0e8a-689f-42cd-9e12-106da73dd5b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c061895-7m288\" (UID: \"bf8f0e8a-689f-42cd-9e12-106da73dd5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c061895-7m288" Oct 30 00:33:10.151944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:10.151895 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c061895-7m288"] Oct 30 00:33:10.255932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:10.248057 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46pfd\" (UniqueName: \"kubernetes.io/projected/bf8f0e8a-689f-42cd-9e12-106da73dd5b1-kube-api-access-46pfd\") pod \"maas-api-revoke-sub-5c061895-7m288\" (UID: \"bf8f0e8a-689f-42cd-9e12-106da73dd5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c061895-7m288" Oct 30 00:33:10.255932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:10.248221 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf8f0e8a-689f-42cd-9e12-106da73dd5b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c061895-7m288\" (UID: \"bf8f0e8a-689f-42cd-9e12-106da73dd5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c061895-7m288" Oct 30 00:33:10.255932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:10.248506 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf8f0e8a-689f-42cd-9e12-106da73dd5b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c061895-7m288\" (UID: \"bf8f0e8a-689f-42cd-9e12-106da73dd5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c061895-7m288" Oct 30 00:33:10.255932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:10.249484 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf8f0e8a-689f-42cd-9e12-106da73dd5b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c061895-7m288\" (UID: \"bf8f0e8a-689f-42cd-9e12-106da73dd5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c061895-7m288" Oct 30 00:33:10.290440 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:10.290400 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf8f0e8a-689f-42cd-9e12-106da73dd5b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c061895-7m288\" (UID: \"bf8f0e8a-689f-42cd-9e12-106da73dd5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c061895-7m288" Oct 30 00:33:10.292356 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:10.292123 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46pfd\" (UniqueName: \"kubernetes.io/projected/bf8f0e8a-689f-42cd-9e12-106da73dd5b1-kube-api-access-46pfd\") pod \"maas-api-revoke-sub-5c061895-7m288\" (UID: \"bf8f0e8a-689f-42cd-9e12-106da73dd5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c061895-7m288" Oct 30 00:33:10.365278 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:10.365189 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e45580f4-lwvdl" Oct 30 00:33:10.365439 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:10.365196 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e45580f4-lwvdl" event={"ID":"df407c8d-1211-43cc-be72-7096f158be35","Type":"ContainerDied","Data":"de13abdff4fe5ae880e39abaf8c2c7189d41abf8b564e2fc546c49ea4854c33f"} Oct 30 00:33:10.365439 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:10.365332 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de13abdff4fe5ae880e39abaf8c2c7189d41abf8b564e2fc546c49ea4854c33f" Oct 30 00:33:10.458936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:10.454422 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c061895-7m288" Oct 30 00:33:10.936461 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:33:10.936431 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf8f0e8a_689f_42cd_9e12_106da73dd5b1.slice/crio-d2544e4374ee3b3ae5669735825e3bd3dc4eab7fc9b128e7dd8b4c638abd9941 WatchSource:0}: Error finding container d2544e4374ee3b3ae5669735825e3bd3dc4eab7fc9b128e7dd8b4c638abd9941: Status 404 returned error can't find the container with id d2544e4374ee3b3ae5669735825e3bd3dc4eab7fc9b128e7dd8b4c638abd9941 Oct 30 00:33:10.937788 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:10.937762 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c061895-7m288"] Oct 30 00:33:11.375834 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:11.375792 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c061895-7m288" event={"ID":"bf8f0e8a-689f-42cd-9e12-106da73dd5b1","Type":"ContainerStarted","Data":"8f8bf97b9dbfe60c70fbc1d12b2ed08bf008960ac45c2ef62a0f7455fc10a76e"} Oct 30 00:33:11.375974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:11.375857 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c061895-7m288" event={"ID":"bf8f0e8a-689f-42cd-9e12-106da73dd5b1","Type":"ContainerStarted","Data":"d2544e4374ee3b3ae5669735825e3bd3dc4eab7fc9b128e7dd8b4c638abd9941"} Oct 30 00:33:11.411817 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:11.411774 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c061895-7m288" podStartSLOduration=1.4117613869999999 podStartE2EDuration="1.411761387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:11.411237976 +0000 UTC m=+1706.616739158" watchObservedRunningTime="2026-10-30 00:33:11.411761387 +0000 UTC m=+1706.617262567" Oct 30 00:33:12.387191 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:12.387158 5001 generic.go:356] "Generic (PLEG): container finished" podID="bf8f0e8a-689f-42cd-9e12-106da73dd5b1" containerID="8f8bf97b9dbfe60c70fbc1d12b2ed08bf008960ac45c2ef62a0f7455fc10a76e" exitCode=0 Oct 30 00:33:12.387618 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:12.387244 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c061895-7m288" event={"ID":"bf8f0e8a-689f-42cd-9e12-106da73dd5b1","Type":"ContainerDied","Data":"8f8bf97b9dbfe60c70fbc1d12b2ed08bf008960ac45c2ef62a0f7455fc10a76e"} Oct 30 00:33:13.322615 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.322576 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb"] Oct 30 00:33:13.328650 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.328622 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:33:13.342029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.342000 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-be1f39-openshift-default-dockercfg-bjdmx\"" Oct 30 00:33:13.358814 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.358786 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb"] Oct 30 00:33:13.426991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.426960 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-istio-data\") pod \"e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:33:13.427426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.427101 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-istio-envoy\") pod \"e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:33:13.427426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.427190 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85331496-4439-48d1-aca8-19dc5cf91bae-istiod-ca-cert\") pod \"e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:33:13.427426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.427308 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lblr5\" (UniqueName: \"kubernetes.io/projected/85331496-4439-48d1-aca8-19dc5cf91bae-kube-api-access-lblr5\") pod \"e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:33:13.427426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.427387 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-credential-socket\") pod \"e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:33:13.427609 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.427526 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-workload-socket\") pod \"e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:33:13.427609 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.427556 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-workload-certs\") pod \"e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:33:13.427727 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.427656 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85331496-4439-48d1-aca8-19dc5cf91bae-istio-token\") pod \"e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:33:13.427788 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.427755 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85331496-4439-48d1-aca8-19dc5cf91bae-istio-podinfo\") pod \"e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:33:13.586021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.541217 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-istio-envoy\") pod \"e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:33:13.586021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.541312 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85331496-4439-48d1-aca8-19dc5cf91bae-istiod-ca-cert\") pod \"e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:33:13.586021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.541353 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lblr5\" (UniqueName: \"kubernetes.io/projected/85331496-4439-48d1-aca8-19dc5cf91bae-kube-api-access-lblr5\") pod \"e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:33:13.586021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.541430 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-credential-socket\") pod \"e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:33:13.586021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.541544 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-workload-socket\") pod \"e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:33:13.586021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.541570 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-workload-certs\") pod \"e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:33:13.586021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.541621 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85331496-4439-48d1-aca8-19dc5cf91bae-istio-token\") pod \"e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:33:13.586021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.541698 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85331496-4439-48d1-aca8-19dc5cf91bae-istio-podinfo\") pod \"e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:33:13.586021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.541781 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-istio-data\") pod \"e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:33:13.586021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.542328 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-istio-data\") pod \"e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:33:13.586021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.561094 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-workload-socket\") pod \"e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:33:13.586021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.564354 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-workload-certs\") pod \"e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:33:13.586021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.580898 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-istio-envoy\") pod \"e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:33:13.586021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.582495 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85331496-4439-48d1-aca8-19dc5cf91bae-istiod-ca-cert\") pod \"e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:33:13.586021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.583122 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-credential-socket\") pod \"e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:33:13.609939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.596245 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85331496-4439-48d1-aca8-19dc5cf91bae-istio-token\") pod \"e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:33:13.690737 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.690065 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85331496-4439-48d1-aca8-19dc5cf91bae-istio-podinfo\") pod \"e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:33:13.693385 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.693356 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lblr5\" (UniqueName: \"kubernetes.io/projected/85331496-4439-48d1-aca8-19dc5cf91bae-kube-api-access-lblr5\") pod \"e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:33:13.804733 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.804346 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c061895-7m288" Oct 30 00:33:13.949326 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.949240 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:33:13.982871 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.982839 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf8f0e8a-689f-42cd-9e12-106da73dd5b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf8f0e8a-689f-42cd-9e12-106da73dd5b1-maas-api-xks-ca\") pod \"bf8f0e8a-689f-42cd-9e12-106da73dd5b1\" (UID: \"bf8f0e8a-689f-42cd-9e12-106da73dd5b1\") " Oct 30 00:33:13.983074 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.983044 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf8f0e8a-689f-42cd-9e12-106da73dd5b1-kube-api-access-46pfd\" (UniqueName: \"kubernetes.io/projected/bf8f0e8a-689f-42cd-9e12-106da73dd5b1-kube-api-access-46pfd\") pod \"bf8f0e8a-689f-42cd-9e12-106da73dd5b1\" (UID: \"bf8f0e8a-689f-42cd-9e12-106da73dd5b1\") " Oct 30 00:33:13.983151 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.983138 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf8f0e8a-689f-42cd-9e12-106da73dd5b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf8f0e8a-689f-42cd-9e12-106da73dd5b1-maas-api-openshift-ca\") pod \"bf8f0e8a-689f-42cd-9e12-106da73dd5b1\" (UID: \"bf8f0e8a-689f-42cd-9e12-106da73dd5b1\") " Oct 30 00:33:13.983552 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.983526 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf8f0e8a-689f-42cd-9e12-106da73dd5b1-maas-api-openshift-ca" pod "bf8f0e8a-689f-42cd-9e12-106da73dd5b1" (UID: "bf8f0e8a-689f-42cd-9e12-106da73dd5b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:33:13.983934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.983900 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf8f0e8a-689f-42cd-9e12-106da73dd5b1-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:33:13.985080 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.985055 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf8f0e8a-689f-42cd-9e12-106da73dd5b1-maas-api-xks-ca" pod "bf8f0e8a-689f-42cd-9e12-106da73dd5b1" (UID: "bf8f0e8a-689f-42cd-9e12-106da73dd5b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:33:13.985646 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:13.985631 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf8f0e8a-689f-42cd-9e12-106da73dd5b1-kube-api-access-46pfd" pod "bf8f0e8a-689f-42cd-9e12-106da73dd5b1" (UID: "bf8f0e8a-689f-42cd-9e12-106da73dd5b1"). InnerVolumeSpecName "kube-api-access-46pfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:33:14.085765 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:14.085731 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf8f0e8a-689f-42cd-9e12-106da73dd5b1-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:33:14.085765 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:14.085768 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46pfd\" (UniqueName: \"kubernetes.io/projected/bf8f0e8a-689f-42cd-9e12-106da73dd5b1-kube-api-access-46pfd\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:33:14.297788 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:14.297757 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb"] Oct 30 00:33:14.298374 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:33:14.298347 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85331496_4439_48d1_aca8_19dc5cf91bae.slice/crio-74cc6485ce61a9589f3ca4d2c7a2243edc993dbe698c157e47714ea1416ed702 WatchSource:0}: Error finding container 74cc6485ce61a9589f3ca4d2c7a2243edc993dbe698c157e47714ea1416ed702: Status 404 returned error can't find the container with id 74cc6485ce61a9589f3ca4d2c7a2243edc993dbe698c157e47714ea1416ed702 Oct 30 00:33:14.301556 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:14.301523 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:33:14.301635 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:14.301594 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:33:14.301635 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:14.301624 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:33:14.426054 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:14.426012 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" event={"ID":"85331496-4439-48d1-aca8-19dc5cf91bae","Type":"ContainerStarted","Data":"3cbe2ffbe68f15f32a979205b3d35ab61a6e5e80e49806a3543017ac7ec0895e"} Oct 30 00:33:14.426259 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:14.426112 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" event={"ID":"85331496-4439-48d1-aca8-19dc5cf91bae","Type":"ContainerStarted","Data":"74cc6485ce61a9589f3ca4d2c7a2243edc993dbe698c157e47714ea1416ed702"} Oct 30 00:33:14.428043 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:14.428019 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c061895-7m288" Oct 30 00:33:14.428568 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:14.428146 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c061895-7m288" event={"ID":"bf8f0e8a-689f-42cd-9e12-106da73dd5b1","Type":"ContainerDied","Data":"d2544e4374ee3b3ae5669735825e3bd3dc4eab7fc9b128e7dd8b4c638abd9941"} Oct 30 00:33:14.428663 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:14.428589 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2544e4374ee3b3ae5669735825e3bd3dc4eab7fc9b128e7dd8b4c638abd9941" Oct 30 00:33:14.475524 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:14.475429 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" podStartSLOduration=1.475412281 podStartE2EDuration="1.475412281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:14.467383938 +0000 UTC m=+1709.672885143" watchObservedRunningTime="2026-10-30 00:33:14.475412281 +0000 UTC m=+1709.680913463" Oct 30 00:33:14.949411 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:14.949336 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:33:15.955636 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:15.955602 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:33:16.474934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.470505 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:33:16.497940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.495934 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:33:16.613578 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.613540 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675"] Oct 30 00:33:16.614637 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.614612 5001 cpu_manager.go:414] "containerMap: removing container" podUID="bf8f0e8a-689f-42cd-9e12-106da73dd5b1" containerName="revoke-keys" Oct 30 00:33:16.614727 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.614641 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf8f0e8a-689f-42cd-9e12-106da73dd5b1" containerName="revoke-keys" Oct 30 00:33:16.614853 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.614840 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf8f0e8a-689f-42cd-9e12-106da73dd5b1" containerName="revoke-keys" Oct 30 00:33:16.620218 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.620190 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:33:16.627971 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.627949 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl"] Oct 30 00:33:16.633597 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.633573 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:33:16.636585 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.636558 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675"] Oct 30 00:33:16.667224 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.667200 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl"] Oct 30 00:33:16.676505 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.676467 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27699192-a6f9-4c03-a63d-e19a122dff3a-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:33:16.680349 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.679121 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:33:16.680349 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.679170 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:33:16.680349 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.679245 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pplkb\" (UniqueName: \"kubernetes.io/projected/27699192-a6f9-4c03-a63d-e19a122dff3a-kube-api-access-pplkb\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:33:16.680349 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.679289 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:33:16.680349 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.679348 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:33:16.680349 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.679374 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:33:16.680349 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.679420 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:33:16.680349 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.679472 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:33:16.680349 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.679525 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:33:16.680349 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.679621 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhdxt\" (UniqueName: \"kubernetes.io/projected/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-kube-api-access-zhdxt\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:33:16.680349 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.679658 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:33:16.680349 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.679682 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27699192-a6f9-4c03-a63d-e19a122dff3a-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:33:16.680349 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.679715 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:33:16.680349 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.679739 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:33:16.680349 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.679823 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27699192-a6f9-4c03-a63d-e19a122dff3a-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:33:16.680349 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.679936 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:33:16.680349 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.679989 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:33:16.781535 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.781500 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pplkb\" (UniqueName: \"kubernetes.io/projected/27699192-a6f9-4c03-a63d-e19a122dff3a-kube-api-access-pplkb\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:33:16.781706 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.781568 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:33:16.781706 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.781632 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:33:16.781706 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.781657 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:33:16.781834 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.781704 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:33:16.781834 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.781749 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:33:16.781834 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.781798 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:33:16.781953 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.781887 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhdxt\" (UniqueName: \"kubernetes.io/projected/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-kube-api-access-zhdxt\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:33:16.781953 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.781938 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:33:16.782020 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.781965 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27699192-a6f9-4c03-a63d-e19a122dff3a-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:33:16.782020 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.781998 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:33:16.782088 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.782023 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:33:16.782088 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.782071 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27699192-a6f9-4c03-a63d-e19a122dff3a-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:33:16.782169 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.782154 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:33:16.782221 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.782206 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:33:16.782267 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.782253 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27699192-a6f9-4c03-a63d-e19a122dff3a-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:33:16.782327 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.782314 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:33:16.782364 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.782352 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:33:16.783507 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.783478 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:33:16.784105 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.784078 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:33:16.784651 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.784624 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:33:16.784901 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.784879 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-credential-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:33:16.785160 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.785139 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:33:16.786141 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.786113 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27699192-a6f9-4c03-a63d-e19a122dff3a-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:33:16.787065 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.787039 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-workload-socket\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:33:16.787310 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.787290 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-workload-certs\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:33:16.789497 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.787762 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-istio-data\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:33:16.789497 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.788390 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istiod-ca-cert\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:33:16.790190 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.790133 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:33:16.790800 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.790389 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:33:16.790800 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.790761 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27699192-a6f9-4c03-a63d-e19a122dff3a-istio-podinfo\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:33:16.800549 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.800517 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-istio-envoy\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:33:16.813416 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.811448 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhdxt\" (UniqueName: \"kubernetes.io/projected/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-kube-api-access-zhdxt\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:33:16.822037 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.821973 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pplkb\" (UniqueName: \"kubernetes.io/projected/27699192-a6f9-4c03-a63d-e19a122dff3a-kube-api-access-pplkb\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:33:16.878938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.853388 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27699192-a6f9-4c03-a63d-e19a122dff3a-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:33:16.878938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.858450 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istio-token\") pod \"e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:33:16.963942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.963060 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:33:16.983418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:16.983392 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:33:17.321341 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:33:17.321313 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27699192_a6f9_4c03_a63d_e19a122dff3a.slice/crio-88ccee7ae347a9a1bdef5dfb73a1b385bed12e575a539eac4232af804026366f WatchSource:0}: Error finding container 88ccee7ae347a9a1bdef5dfb73a1b385bed12e575a539eac4232af804026366f: Status 404 returned error can't find the container with id 88ccee7ae347a9a1bdef5dfb73a1b385bed12e575a539eac4232af804026366f Oct 30 00:33:17.324559 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:17.324534 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl"] Oct 30 00:33:17.324685 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:33:17.324600 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:33:17.325688 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:17.325656 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:33:17.325792 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:17.325715 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:33:17.325792 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:17.325785 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:33:17.330819 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:17.330797 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675"] Oct 30 00:33:17.332288 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:33:17.332264 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5e89f70_0dab_4ed5_b01b_2e78feab8b8e.slice/crio-c62e4a888861a39680d5b94a2b980e27ab10f267e055454e8f0bfe1b72e0787f WatchSource:0}: Error finding container c62e4a888861a39680d5b94a2b980e27ab10f267e055454e8f0bfe1b72e0787f: Status 404 returned error can't find the container with id c62e4a888861a39680d5b94a2b980e27ab10f267e055454e8f0bfe1b72e0787f Oct 30 00:33:17.339631 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:17.339586 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:33:17.339772 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:17.339745 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:33:17.339864 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:17.339846 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:33:17.521550 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:17.521421 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" event={"ID":"27699192-a6f9-4c03-a63d-e19a122dff3a","Type":"ContainerStarted","Data":"d18662b9f840e4fd76419aec628dc1a54abbc78905d9218d8ce115c3c38a8593"} Oct 30 00:33:17.521888 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:17.521825 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" event={"ID":"27699192-a6f9-4c03-a63d-e19a122dff3a","Type":"ContainerStarted","Data":"88ccee7ae347a9a1bdef5dfb73a1b385bed12e575a539eac4232af804026366f"} Oct 30 00:33:17.526308 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:17.526264 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" event={"ID":"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e","Type":"ContainerStarted","Data":"e42b653a6ed631241bde2d9d65f58d134b03524efc2d224f5e8766e5fff9044c"} Oct 30 00:33:17.526514 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:17.526478 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" event={"ID":"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e","Type":"ContainerStarted","Data":"c62e4a888861a39680d5b94a2b980e27ab10f267e055454e8f0bfe1b72e0787f"} Oct 30 00:33:17.585888 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:17.585713 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" podStartSLOduration=1.585678282 podStartE2EDuration="1.585678282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:17.584245002 +0000 UTC m=+1712.789746184" watchObservedRunningTime="2026-10-30 00:33:17.585678282 +0000 UTC m=+1712.791179464" Oct 30 00:33:17.638488 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:17.635864 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" podStartSLOduration=1.6358429349999999 podStartE2EDuration="1.635842935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:17.630481931 +0000 UTC m=+1712.835983111" watchObservedRunningTime="2026-10-30 00:33:17.635842935 +0000 UTC m=+1712.841344120" Oct 30 00:33:17.859800 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:17.859711 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m"] Oct 30 00:33:17.865850 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:17.865821 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:33:17.886512 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:17.886487 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m"] Oct 30 00:33:17.931748 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:17.931711 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44355452-926f-4608-b0eb-24bb7e30dcbd-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:33:17.932011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:17.931809 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:33:17.932011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:17.931898 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:33:17.932011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:17.932006 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44355452-926f-4608-b0eb-24bb7e30dcbd-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:33:17.932221 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:17.932036 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:33:17.932221 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:17.932114 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pklw7\" (UniqueName: \"kubernetes.io/projected/44355452-926f-4608-b0eb-24bb7e30dcbd-kube-api-access-pklw7\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:33:17.932221 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:17.932141 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:33:17.932391 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:17.932266 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:33:17.932391 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:17.932336 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44355452-926f-4608-b0eb-24bb7e30dcbd-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:33:17.963927 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:17.963890 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:33:17.984077 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:17.984047 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:33:18.033802 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:18.033765 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44355452-926f-4608-b0eb-24bb7e30dcbd-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:33:18.034015 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:18.033846 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:33:18.034015 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:18.033971 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:33:18.034341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:18.034305 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:33:18.034483 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:18.034365 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:33:18.034483 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:18.034432 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44355452-926f-4608-b0eb-24bb7e30dcbd-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:33:18.034483 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:18.034463 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:33:18.034776 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:18.034523 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pklw7\" (UniqueName: \"kubernetes.io/projected/44355452-926f-4608-b0eb-24bb7e30dcbd-kube-api-access-pklw7\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:33:18.034776 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:18.034556 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:33:18.034776 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:18.034647 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:33:18.034776 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:18.034706 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44355452-926f-4608-b0eb-24bb7e30dcbd-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:33:18.036635 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:18.035506 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44355452-926f-4608-b0eb-24bb7e30dcbd-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:33:18.036635 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:18.035781 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:33:18.036635 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:18.036060 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:33:18.037456 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:18.037432 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44355452-926f-4608-b0eb-24bb7e30dcbd-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:33:18.037725 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:18.037707 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:33:18.055972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:18.055901 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44355452-926f-4608-b0eb-24bb7e30dcbd-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:33:18.056579 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:18.056551 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pklw7\" (UniqueName: \"kubernetes.io/projected/44355452-926f-4608-b0eb-24bb7e30dcbd-kube-api-access-pklw7\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:33:18.188999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:18.188813 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:33:18.636944 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:33:18.635448 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44355452_926f_4608_b0eb_24bb7e30dcbd.slice/crio-1bada394f033b49e6c1f5166bd4ae2cfd0a6da4a3345b3089cb8d64364f6dbd1 WatchSource:0}: Error finding container 1bada394f033b49e6c1f5166bd4ae2cfd0a6da4a3345b3089cb8d64364f6dbd1: Status 404 returned error can't find the container with id 1bada394f033b49e6c1f5166bd4ae2cfd0a6da4a3345b3089cb8d64364f6dbd1 Oct 30 00:33:18.649079 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:18.649034 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:33:18.649312 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:18.649289 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:33:18.649470 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:18.649452 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:33:18.664967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:18.664266 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m"] Oct 30 00:33:18.968993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:18.968091 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 30 00:33:18.968993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:18.968152 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" podUID="b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 30 00:33:18.985237 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:18.985206 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 00:33:18.985609 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:18.985258 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" podUID="27699192-a6f9-4c03-a63d-e19a122dff3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 00:33:19.590753 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:19.589187 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" event={"ID":"44355452-926f-4608-b0eb-24bb7e30dcbd","Type":"ContainerStarted","Data":"379b761ea39da3c3064a149b8cc39298798c811cf0ad42d8d67857184eedade5"} Oct 30 00:33:19.591264 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:19.591235 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" event={"ID":"44355452-926f-4608-b0eb-24bb7e30dcbd","Type":"ContainerStarted","Data":"1bada394f033b49e6c1f5166bd4ae2cfd0a6da4a3345b3089cb8d64364f6dbd1"} Oct 30 00:33:19.772939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:19.717968 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" podStartSLOduration=2.717946973 podStartE2EDuration="2.717946973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:19.716304336 +0000 UTC m=+1714.921805518" watchObservedRunningTime="2026-10-30 00:33:19.717946973 +0000 UTC m=+1714.923448155" Oct 30 00:33:19.964508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:19.964414 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 30 00:33:19.966019 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:19.964715 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" podUID="b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 30 00:33:20.072852 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:20.072816 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 00:33:20.073496 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:20.073461 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" podUID="27699192-a6f9-4c03-a63d-e19a122dff3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 00:33:20.193123 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:20.192853 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 00:33:20.193419 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:20.193374 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" podUID="44355452-926f-4608-b0eb-24bb7e30dcbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 00:33:20.193897 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:20.193878 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:33:20.969973 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:20.969934 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 30 00:33:20.970256 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:20.970227 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" podUID="b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 30 00:33:20.990521 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:20.990380 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 00:33:20.990521 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:20.990455 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" podUID="27699192-a6f9-4c03-a63d-e19a122dff3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 00:33:21.192395 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:21.192355 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 00:33:21.193065 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:21.193019 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" podUID="44355452-926f-4608-b0eb-24bb7e30dcbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 00:33:21.964937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:21.964502 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 30 00:33:21.964937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:21.964563 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" podUID="b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 30 00:33:22.001983 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:22.001275 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 00:33:22.001983 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:22.001350 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" podUID="27699192-a6f9-4c03-a63d-e19a122dff3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 00:33:22.190788 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:22.190754 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 00:33:22.191081 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:22.190810 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" podUID="44355452-926f-4608-b0eb-24bb7e30dcbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 00:33:22.963671 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:22.963634 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 30 00:33:22.964311 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:22.964257 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" podUID="b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 30 00:33:22.998238 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:22.998201 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 00:33:22.998556 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:22.998510 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" podUID="27699192-a6f9-4c03-a63d-e19a122dff3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 00:33:23.198940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:23.197389 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 00:33:23.198940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:23.197454 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" podUID="44355452-926f-4608-b0eb-24bb7e30dcbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 00:33:23.982881 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:23.982842 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 30 00:33:23.983737 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:23.983707 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" podUID="b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 30 00:33:23.984261 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:23.983647 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 00:33:23.984430 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:23.984408 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" podUID="27699192-a6f9-4c03-a63d-e19a122dff3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 00:33:24.199355 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:24.199316 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 00:33:24.199660 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:24.199628 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" podUID="44355452-926f-4608-b0eb-24bb7e30dcbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 00:33:24.896238 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:24.896192 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:25.126935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:24.976937 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 30 00:33:25.126935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:24.976992 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" podUID="b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 30 00:33:25.126935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:25.007936 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 00:33:25.126935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:25.007992 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" podUID="27699192-a6f9-4c03-a63d-e19a122dff3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 00:33:25.126935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:25.061183 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:25.126935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:25.073046 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:25.256147 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:25.256056 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 00:33:25.256421 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:25.256387 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" podUID="44355452-926f-4608-b0eb-24bb7e30dcbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 00:33:25.384051 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:25.384001 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:25.517948 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:25.516600 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:25.910179 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:25.910069 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:25.956109 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:25.956052 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:33:25.957367 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:25.957345 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:33:25.980936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:25.980883 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 30 00:33:25.981209 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:25.981181 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" podUID="b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 30 00:33:25.993406 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:25.993368 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 00:33:25.993657 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:25.993629 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" podUID="27699192-a6f9-4c03-a63d-e19a122dff3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 00:33:26.200859 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:26.192677 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 00:33:26.392964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:26.392921 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" podUID="44355452-926f-4608-b0eb-24bb7e30dcbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 00:33:26.702937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:26.242220 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:33:26.721938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:26.242256 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 30 00:33:26.721938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:26.704031 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:33:26.722951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:26.704124 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:33:26.727010 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:26.559600 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:26.727010 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:26.726165 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:33:26.742359 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:26.242158 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:33:26.744344 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:26.393318 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:26.749351 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:26.745022 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:33:26.920474 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:26.920423 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:26.961997 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:26.958002 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:33:27.072937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:27.047776 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 00:33:27.531951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:27.528013 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" podUID="27699192-a6f9-4c03-a63d-e19a122dff3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 00:33:27.531951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:27.336057 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 00:33:27.531951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:27.122708 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:27.531951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:27.073683 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 30 00:33:27.531951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:27.529232 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" podUID="b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 30 00:33:27.531951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:27.342026 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 00:33:27.531951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:27.529278 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" podUID="44355452-926f-4608-b0eb-24bb7e30dcbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 00:33:27.531951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:27.407111 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:33:27.735938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:27.720031 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 30 00:33:27.735938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:27.721839 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:33:27.760952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:27.751007 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:27.790032 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:27.781727 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:27.951791 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:27.951741 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:27.952586 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:27.952553 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:28.039940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:28.029760 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:33:28.116251 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:28.116154 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 00:33:28.495526 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:28.495427 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" podUID="27699192-a6f9-4c03-a63d-e19a122dff3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 00:33:28.496789 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:28.223855 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7584-61fb81"] Oct 30 00:33:28.529016 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:28.467399 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": context deadline exceeded" Oct 30 00:33:28.529327 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:28.149826 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:28.529887 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:28.223786 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 00:33:28.547937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:28.530083 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" podUID="44355452-926f-4608-b0eb-24bb7e30dcbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 00:33:28.549506 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:28.306762 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 30 00:33:28.549749 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:28.549699 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" podUID="b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 30 00:33:28.550701 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:28.550671 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 00:33:28.648492 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:28.648460 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:33:28.650041 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:28.650018 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7584-61fb81" Oct 30 00:33:28.725601 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:28.725561 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hr9mw\"" Oct 30 00:33:28.798999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:28.752384 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:28.871961 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:28.777407 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:28.871961 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:28.777459 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 00:33:28.871961 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:28.812209 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4cxl\" (UniqueName: \"kubernetes.io/projected/9b197a22-4dd0-4d03-8379-488fa2ad6a5e-kube-api-access-t4cxl\") pod \"test-curl-7584-61fb81\" (UID: \"9b197a22-4dd0-4d03-8379-488fa2ad6a5e\") " pod="opendatahub/test-curl-7584-61fb81" Oct 30 00:33:28.871961 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:28.812353 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 00:33:28.905083 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:28.905052 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7584-61fb81"] Oct 30 00:33:28.946944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:28.921268 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:28.974434 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:28.974398 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4cxl\" (UniqueName: \"kubernetes.io/projected/9b197a22-4dd0-4d03-8379-488fa2ad6a5e-kube-api-access-t4cxl\") pod \"test-curl-7584-61fb81\" (UID: \"9b197a22-4dd0-4d03-8379-488fa2ad6a5e\") " pod="opendatahub/test-curl-7584-61fb81" Oct 30 00:33:29.003060 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:28.975531 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:29.003646 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:28.977166 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:29.004341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:28.977482 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 30 00:33:29.013994 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:29.009065 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" podUID="b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 30 00:33:29.294035 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:29.209136 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 00:33:29.297428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:29.256651 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 00:33:29.297676 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:29.297647 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" podUID="27699192-a6f9-4c03-a63d-e19a122dff3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 00:33:29.297836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:29.295403 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:33:29.298023 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:29.297970 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" podUID="cf513d72-64d7-4444-ae0c-fda3aef69626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:29.412008 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:29.306008 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" podUID="44355452-926f-4608-b0eb-24bb7e30dcbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 00:33:29.412266 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:29.306486 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:29.570988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:29.494927 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:33:29.610107 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:29.511814 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4cxl\" (UniqueName: \"kubernetes.io/projected/9b197a22-4dd0-4d03-8379-488fa2ad6a5e-kube-api-access-t4cxl\") pod \"test-curl-7584-61fb81\" (UID: \"9b197a22-4dd0-4d03-8379-488fa2ad6a5e\") " pod="opendatahub/test-curl-7584-61fb81" Oct 30 00:33:29.618013 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:29.530506 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:33:29.670027 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:29.610992 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:29.677353 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:29.670434 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" podUID="ea34fb77-6b08-4939-b1c0-9ba33e8afd0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:29.724208 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:29.724086 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7584-61fb81" Oct 30 00:33:29.898935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:29.886519 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:33:29.991939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:29.898006 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:29.991939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:29.898066 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 00:33:29.991939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:29.898104 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 30 00:33:29.991939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:29.966854 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 30 00:33:29.991939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:29.979506 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" podUID="b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 30 00:33:30.017511 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:30.015571 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 00:33:30.017511 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:30.015627 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" podUID="27699192-a6f9-4c03-a63d-e19a122dff3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 00:33:30.017511 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:30.015693 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:33:30.017511 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:30.016007 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:33:30.211535 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:30.211434 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 00:33:30.211824 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:30.211777 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" podUID="44355452-926f-4608-b0eb-24bb7e30dcbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 00:33:30.618879 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:30.618829 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:30.629565 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:30.629519 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:30.993002 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:30.982050 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 00:33:31.026942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:30.986793 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:33:31.039943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:30.997439 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:31.044276 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:31.029331 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:31.046305 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:31.029369 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:33:31.050569 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:31.032708 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:31.124945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:31.123635 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 30 00:33:31.124945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:31.123699 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" podUID="b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 30 00:33:31.213954 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:31.146960 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 00:33:31.213954 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:31.147020 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" podUID="27699192-a6f9-4c03-a63d-e19a122dff3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 00:33:31.213954 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:31.194088 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 00:33:31.213954 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:31.194153 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" podUID="44355452-926f-4608-b0eb-24bb7e30dcbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 00:33:31.511814 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:31.510610 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:33:31.511814 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:31.510684 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" podUID="6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:32.040993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:32.007060 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:33:32.065938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:32.045025 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:33:32.106937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:32.072196 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 00:33:32.107344 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:32.107305 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:33:32.108270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:32.108239 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:33:32.109075 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:32.109048 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 00:33:32.156591 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:32.106684 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 00:33:32.157996 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:32.106877 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:32.158491 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:32.107095 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:32.203668 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:32.188129 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded" Oct 30 00:33:32.306167 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:32.106463 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": context deadline exceeded" Oct 30 00:33:32.320667 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:32.254066 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:33:32.459251 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:32.279443 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 30 00:33:32.459251 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:32.459037 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" podUID="b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 30 00:33:32.494939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:32.394382 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 00:33:32.495297 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:32.495253 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" podUID="27699192-a6f9-4c03-a63d-e19a122dff3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 00:33:32.528957 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:32.525394 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 00:33:32.528957 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:32.525649 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" podUID="44355452-926f-4608-b0eb-24bb7e30dcbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 00:33:33.095974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:33.083201 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:33:33.115192 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:33.115133 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:33.228064 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:33.227993 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:33.286956 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:33.279534 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:33.286956 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:33.279964 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:33.292073 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:33.287643 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:33:33.307000 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:33.305297 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:33:33.307000 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:33.305360 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" podUID="ad5882d9-90e8-400f-90d5-d4ba5d8da5fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:33.307000 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:33.305535 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:33:33.308121 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:33.308090 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 00:33:33.331232 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:33.330246 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:33.348317 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:33.348204 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" podUID="d5dc33b0-82db-44f6-90df-dabe7979e9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:33.350099 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:33.349494 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:33:33.350099 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:33.349556 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" podUID="1450166b-4504-4cb0-adac-c43c514a06fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:33.502332 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:33.502293 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 00:33:33.502724 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:33.502687 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" podUID="44355452-926f-4608-b0eb-24bb7e30dcbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 00:33:33.502947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:33.502925 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 30 00:33:33.503097 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:33.503076 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" podUID="b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 30 00:33:33.850335 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:33.850284 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 00:33:33.964531 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:33.964440 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:33:33.985971 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:33.985944 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1c718-mjsnx"] Oct 30 00:33:34.106972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:33.854357 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" podUID="27699192-a6f9-4c03-a63d-e19a122dff3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 00:33:34.107868 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.040032 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1c718-mjsnx" Oct 30 00:33:34.140230 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.070258 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1c718-mjsnx"] Oct 30 00:33:34.140520 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.073326 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 30 00:33:34.140708 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.140681 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" podUID="b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 30 00:33:34.140831 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.094233 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:34.141589 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.094152 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:33:34.141974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.131015 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 00:33:34.154053 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.154001 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" podUID="27699192-a6f9-4c03-a63d-e19a122dff3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 00:33:34.169533 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.169477 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 00:33:34.180812 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.180777 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf"] Oct 30 00:33:34.281082 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.281030 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:34.281617 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.281153 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:34.290946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.281194 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:33:34.441938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.300345 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc4vj\" (UniqueName: \"kubernetes.io/projected/89e9208c-ef2c-483d-a9bb-666416fd9076-kube-api-access-kc4vj\") pod \"maas-api-revoke-keys-e2e-worker-w5-b1c718-mjsnx\" (UID: \"89e9208c-ef2c-483d-a9bb-666416fd9076\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1c718-mjsnx" Oct 30 00:33:34.442428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.442390 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e9208c-ef2c-483d-a9bb-666416fd9076-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-b1c718-mjsnx\" (UID: \"89e9208c-ef2c-483d-a9bb-666416fd9076\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1c718-mjsnx" Oct 30 00:33:34.442685 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.442663 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e9208c-ef2c-483d-a9bb-666416fd9076-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-b1c718-mjsnx\" (UID: \"89e9208c-ef2c-483d-a9bb-666416fd9076\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1c718-mjsnx" Oct 30 00:33:34.455079 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.300452 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:33:34.464939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.373643 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:33:34.489990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.374452 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:33:34.490893 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.374518 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:33:34.491085 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.374576 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:34.491503 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.340483 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 00:33:34.491686 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.491657 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" podUID="44355452-926f-4608-b0eb-24bb7e30dcbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 00:33:34.491812 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.446046 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:33:34.521015 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.520974 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" podUID="b2cdf918-8248-4079-9823-830a84e927bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:34.521267 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.343360 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:33:34.554788 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.536866 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:33:34.626043 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.626000 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" podUID="f549cc40-a1ae-43aa-825f-11d538b9fa71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:34.626587 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.556531 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:33:34.626796 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.626781 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf"] Oct 30 00:33:34.626984 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.553231 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e9208c-ef2c-483d-a9bb-666416fd9076-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-b1c718-mjsnx\" (UID: \"89e9208c-ef2c-483d-a9bb-666416fd9076\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1c718-mjsnx" Oct 30 00:33:34.627430 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.627409 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc4vj\" (UniqueName: \"kubernetes.io/projected/89e9208c-ef2c-483d-a9bb-666416fd9076-kube-api-access-kc4vj\") pod \"maas-api-revoke-keys-e2e-worker-w5-b1c718-mjsnx\" (UID: \"89e9208c-ef2c-483d-a9bb-666416fd9076\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1c718-mjsnx" Oct 30 00:33:34.671161 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.671125 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e9208c-ef2c-483d-a9bb-666416fd9076-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-b1c718-mjsnx\" (UID: \"89e9208c-ef2c-483d-a9bb-666416fd9076\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1c718-mjsnx" Oct 30 00:33:34.671530 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.667126 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:33:34.677956 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.643443 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e9208c-ef2c-483d-a9bb-666416fd9076-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-b1c718-mjsnx\" (UID: \"89e9208c-ef2c-483d-a9bb-666416fd9076\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1c718-mjsnx" Oct 30 00:33:34.681934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.678755 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e9208c-ef2c-483d-a9bb-666416fd9076-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-b1c718-mjsnx\" (UID: \"89e9208c-ef2c-483d-a9bb-666416fd9076\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1c718-mjsnx" Oct 30 00:33:34.704154 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.668194 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7584-61fb81"] Oct 30 00:33:34.704501 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.704479 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:33:34.788941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.786599 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:33:34.788941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.786846 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1222e77c-6ae5-4ce1-a15f-caa491260730-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:33:34.788941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.786937 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1222e77c-6ae5-4ce1-a15f-caa491260730-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:33:34.788941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.787031 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:33:34.788941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.787059 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:33:34.788941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.787173 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:33:34.788941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.787283 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1222e77c-6ae5-4ce1-a15f-caa491260730-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:33:34.788941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.787317 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:33:34.788941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.787357 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqhvf\" (UniqueName: \"kubernetes.io/projected/1222e77c-6ae5-4ce1-a15f-caa491260730-kube-api-access-nqhvf\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:33:34.790357 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.790313 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:34.795519 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.795491 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7584-61fb81" event={"ID":"9b197a22-4dd0-4d03-8379-488fa2ad6a5e","Type":"ContainerStarted","Data":"be1f001d6f608123ea964ad4c20191d7b131b68aa2942eef651813278f6055f9"} Oct 30 00:33:34.913965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.899117 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1222e77c-6ae5-4ce1-a15f-caa491260730-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:33:34.913965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.899219 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:33:34.913965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.899247 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:33:34.913965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.899343 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:33:34.913965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.899442 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1222e77c-6ae5-4ce1-a15f-caa491260730-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:33:34.913965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.899479 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:33:34.913965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.899519 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqhvf\" (UniqueName: \"kubernetes.io/projected/1222e77c-6ae5-4ce1-a15f-caa491260730-kube-api-access-nqhvf\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:33:34.913965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.899576 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:33:34.913965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.899821 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1222e77c-6ae5-4ce1-a15f-caa491260730-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:33:34.922962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.914890 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1222e77c-6ae5-4ce1-a15f-caa491260730-istiod-ca-cert\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:33:34.970710 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:34.928266 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-workload-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:33:35.063944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:35.001408 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-workload-certs\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:33:35.063944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:35.002028 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-credential-socket\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:33:35.063944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:35.002596 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1222e77c-6ae5-4ce1-a15f-caa491260730-istio-podinfo\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:33:35.098506 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:35.027436 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-istio-data\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:33:35.098506 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:35.027796 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 30 00:33:35.098506 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:35.078286 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" podUID="b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 30 00:33:35.098506 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:35.091353 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 00:33:35.098506 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:35.091429 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" podUID="27699192-a6f9-4c03-a63d-e19a122dff3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 00:33:35.154782 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:35.144631 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:35.261937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:35.233012 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 00:33:35.301127 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:35.301082 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 00:33:35.301416 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:35.301374 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" podUID="44355452-926f-4608-b0eb-24bb7e30dcbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 00:33:35.302079 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:35.302038 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:35.324463 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:35.324355 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:35.335960 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:35.331657 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 30 00:33:35.360011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:35.357951 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:33:35.399938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:35.382765 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc4vj\" (UniqueName: \"kubernetes.io/projected/89e9208c-ef2c-483d-a9bb-666416fd9076-kube-api-access-kc4vj\") pod \"maas-api-revoke-keys-e2e-worker-w5-b1c718-mjsnx\" (UID: \"89e9208c-ef2c-483d-a9bb-666416fd9076\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1c718-mjsnx" Oct 30 00:33:35.400205 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:35.390836 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85"] Oct 30 00:33:35.424549 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:35.391743 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-istio-envoy\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:33:35.500980 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:35.492015 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:33:35.643015 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:35.624364 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1c718-mjsnx" Oct 30 00:33:35.643015 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:35.625252 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:35.677728 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:35.677657 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqhvf\" (UniqueName: \"kubernetes.io/projected/1222e77c-6ae5-4ce1-a15f-caa491260730-kube-api-access-nqhvf\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:33:35.678905 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:35.678876 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1222e77c-6ae5-4ce1-a15f-caa491260730-istio-token\") pod \"e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:33:35.831663 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:35.831396 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:33:36.316936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:36.311423 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 00:33:36.316936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:36.311498 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" podUID="44355452-926f-4608-b0eb-24bb7e30dcbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 00:33:36.316936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:36.312724 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:33:36.362982 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:36.341608 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 30 00:33:36.409123 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:36.405589 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 30 00:33:36.633866 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:36.633821 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" podUID="b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 30 00:33:36.713109 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:36.529466 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:36.713109 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:36.530062 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 00:33:36.713474 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:36.552693 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:36.719973 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:36.585778 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 00:33:36.719973 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:36.514073 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ecb34a-7ffdbf96c-8lgp2"] Oct 30 00:33:36.735433 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:36.735386 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:36.754171 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:36.662765 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:36.768397 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:36.528481 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 00:33:36.768397 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:36.763014 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" podUID="27699192-a6f9-4c03-a63d-e19a122dff3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 00:33:37.227043 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:37.217966 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 00:33:37.227043 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:37.218028 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" podUID="44355452-926f-4608-b0eb-24bb7e30dcbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 00:33:37.295594 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:37.295531 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:37.320659 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:37.320016 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:37.354972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:37.351054 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:37.761650 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:37.715932 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:38.088386 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:37.729547 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:38.104545 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:37.729588 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 00:33:38.105186 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:37.729636 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:33:38.105751 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:37.787486 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ecb34a-7ffdbf96c-8lgp2" Oct 30 00:33:38.106573 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:37.787560 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:33:38.124935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:37.787611 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:38.128096 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:37.930239 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca30d4b-8350-4bfd-ba90-54e70dacf765-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-ecb34a-7ffdbf96c-8lgp2\" (UID: \"aca30d4b-8350-4bfd-ba90-54e70dacf765\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ecb34a-7ffdbf96c-8lgp2" Oct 30 00:33:38.128382 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:38.128345 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbvr7\" (UniqueName: \"kubernetes.io/projected/aca30d4b-8350-4bfd-ba90-54e70dacf765-kube-api-access-lbvr7\") pod \"payload-pre-processing-e2e-shared-b-w4-ecb34a-7ffdbf96c-8lgp2\" (UID: \"aca30d4b-8350-4bfd-ba90-54e70dacf765\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ecb34a-7ffdbf96c-8lgp2" Oct 30 00:33:38.130437 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:38.130412 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:33:38.130644 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:38.130619 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" podUID="27699192-a6f9-4c03-a63d-e19a122dff3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:38.185222 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:38.185186 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:33:38.185482 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:38.185454 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" podUID="b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:38.365776 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:38.325469 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:38.365776 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:38.326236 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-ecb34a-dockercfg-djlh9\"" Oct 30 00:33:38.365776 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:38.326459 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-ecb34a\"" Oct 30 00:33:38.365776 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:38.343194 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 00:33:38.365776 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:38.343259 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" podUID="44355452-926f-4608-b0eb-24bb7e30dcbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 00:33:38.382423 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:38.359086 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:38.392406 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:38.382882 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca30d4b-8350-4bfd-ba90-54e70dacf765-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-ecb34a-7ffdbf96c-8lgp2\" (UID: \"aca30d4b-8350-4bfd-ba90-54e70dacf765\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ecb34a-7ffdbf96c-8lgp2" Oct 30 00:33:38.392406 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:38.382952 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbvr7\" (UniqueName: \"kubernetes.io/projected/aca30d4b-8350-4bfd-ba90-54e70dacf765-kube-api-access-lbvr7\") pod \"payload-pre-processing-e2e-shared-b-w4-ecb34a-7ffdbf96c-8lgp2\" (UID: \"aca30d4b-8350-4bfd-ba90-54e70dacf765\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ecb34a-7ffdbf96c-8lgp2" Oct 30 00:33:38.429930 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:38.424904 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca30d4b-8350-4bfd-ba90-54e70dacf765-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-ecb34a-7ffdbf96c-8lgp2\" (UID: \"aca30d4b-8350-4bfd-ba90-54e70dacf765\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ecb34a-7ffdbf96c-8lgp2" Oct 30 00:33:38.570956 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:38.558951 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ecb34a-7ffdbf96c-8lgp2"] Oct 30 00:33:38.570956 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:38.559050 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:33:38.570956 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:38.559073 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-ecb34a-7c4f79cf86-6l8nz"] Oct 30 00:33:38.791134 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:38.791086 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:33:38.808178 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:38.808135 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:33:38.808460 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:38.808437 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:33:38.809040 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:38.801054 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:33:38.822972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:38.801401 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ecb34a-7c4f79cf86-6l8nz" Oct 30 00:33:38.823237 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:38.823218 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-ecb34a-7c4f79cf86-6l8nz"] Oct 30 00:33:38.823385 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:38.823372 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1c718-mjsnx"] Oct 30 00:33:38.905242 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:38.905154 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Oct 30 00:33:39.098947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:39.097311 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0261228f-3733-4993-893e-43c742c7bd5e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-ecb34a-7c4f79cf86-6l8nz\" (UID: \"0261228f-3733-4993-893e-43c742c7bd5e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ecb34a-7c4f79cf86-6l8nz" Oct 30 00:33:39.098947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:39.097448 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7d7g\" (UniqueName: \"kubernetes.io/projected/0261228f-3733-4993-893e-43c742c7bd5e-kube-api-access-v7d7g\") pod \"payload-processing-e2e-shared-b-w4-ecb34a-7c4f79cf86-6l8nz\" (UID: \"0261228f-3733-4993-893e-43c742c7bd5e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ecb34a-7c4f79cf86-6l8nz" Oct 30 00:33:39.099199 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:39.098962 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:39.123050 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:39.099927 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:39.123050 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:39.105436 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:39.123050 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:39.105732 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:39.123050 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:39.106863 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:39.163983 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:39.129315 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:39.163983 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:39.139071 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:33:39.163983 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:39.139126 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" podUID="27699192-a6f9-4c03-a63d-e19a122dff3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:39.432806 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:39.301683 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 00:33:39.432806 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:39.301741 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" podUID="44355452-926f-4608-b0eb-24bb7e30dcbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 00:33:39.509873 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:39.509829 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:39.536196 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:39.536154 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:39.547941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:39.543814 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbvr7\" (UniqueName: \"kubernetes.io/projected/aca30d4b-8350-4bfd-ba90-54e70dacf765-kube-api-access-lbvr7\") pod \"payload-pre-processing-e2e-shared-b-w4-ecb34a-7ffdbf96c-8lgp2\" (UID: \"aca30d4b-8350-4bfd-ba90-54e70dacf765\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ecb34a-7ffdbf96c-8lgp2" Oct 30 00:33:39.596442 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:39.564885 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:39.596442 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:39.571940 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0261228f-3733-4993-893e-43c742c7bd5e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-ecb34a-7c4f79cf86-6l8nz\" (UID: \"0261228f-3733-4993-893e-43c742c7bd5e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ecb34a-7c4f79cf86-6l8nz" Oct 30 00:33:39.598051 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:39.598015 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:33:39.626904 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:39.626845 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" podUID="b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:39.688439 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:39.607740 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ecb34a-7ffdbf96c-8lgp2" Oct 30 00:33:39.705294 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:39.704201 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0261228f-3733-4993-893e-43c742c7bd5e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-ecb34a-7c4f79cf86-6l8nz\" (UID: \"0261228f-3733-4993-893e-43c742c7bd5e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ecb34a-7c4f79cf86-6l8nz" Oct 30 00:33:39.744172 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:39.744133 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7d7g\" (UniqueName: \"kubernetes.io/projected/0261228f-3733-4993-893e-43c742c7bd5e-kube-api-access-v7d7g\") pod \"payload-processing-e2e-shared-b-w4-ecb34a-7c4f79cf86-6l8nz\" (UID: \"0261228f-3733-4993-893e-43c742c7bd5e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ecb34a-7c4f79cf86-6l8nz" Oct 30 00:33:39.861835 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:39.861803 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7d7g\" (UniqueName: \"kubernetes.io/projected/0261228f-3733-4993-893e-43c742c7bd5e-kube-api-access-v7d7g\") pod \"payload-processing-e2e-shared-b-w4-ecb34a-7c4f79cf86-6l8nz\" (UID: \"0261228f-3733-4993-893e-43c742c7bd5e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ecb34a-7c4f79cf86-6l8nz" Oct 30 00:33:40.108335 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:40.108285 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:40.110004 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:40.109971 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 00:33:40.111707 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:40.111685 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ecb34a-7c4f79cf86-6l8nz" Oct 30 00:33:40.150934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:40.139090 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:33:40.150934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:40.139971 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 00:33:40.165090 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:40.152178 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:33:40.166431 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:40.165359 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:33:40.166431 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:40.165428 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" podUID="27699192-a6f9-4c03-a63d-e19a122dff3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:40.281079 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:40.281036 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:40.356940 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:33:40.347409 5001 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.432s" Oct 30 00:33:40.477357 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:40.477309 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7584-61fb81" event={"ID":"9b197a22-4dd0-4d03-8379-488fa2ad6a5e","Type":"ContainerStarted","Data":"38bc6e344ce6fe603763cdfaf2ae897744f809e4de943fdf630028be974bede6"} Oct 30 00:33:40.477357 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:40.477361 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1c718-mjsnx" event={"ID":"89e9208c-ef2c-483d-a9bb-666416fd9076","Type":"ContainerStarted","Data":"ef30773f220627929e62093cf7836d2e11532fb5ca0152d8df86c61cf2569085"} Oct 30 00:33:40.477950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:40.477408 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf"] Oct 30 00:33:40.488015 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:33:40.482210 5001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca30d4b_8350_4bfd_ba90_54e70dacf765.slice\": RecentStats: unable to find data in memory cache]" Oct 30 00:33:40.581939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:40.554857 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:33:40.581939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:40.559637 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:40.590186 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:40.590141 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:40.637938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:40.616213 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" event={"ID":"1222e77c-6ae5-4ce1-a15f-caa491260730","Type":"ContainerStarted","Data":"d27613a0eb46b85979916787ac5d9994961123d7104be2bc048ebe517f9d158d"} Oct 30 00:33:40.788962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:40.723031 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:33:40.788962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:40.780062 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" podUID="b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:41.289936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:41.271200 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:41.423490 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:41.410982 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:41.423490 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:41.412427 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 00:33:41.444488 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:41.436767 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:33:41.474935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:41.471264 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:41.521521 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:41.501623 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:33:41.521521 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:41.501853 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" podUID="44355452-926f-4608-b0eb-24bb7e30dcbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:41.521521 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:41.502162 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:33:41.521521 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:41.502201 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" podUID="27699192-a6f9-4c03-a63d-e19a122dff3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:41.805732 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:41.786038 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:41.814725 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:41.814661 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:41.861035 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:41.860257 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:33:41.861035 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:41.860334 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" podUID="b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:42.418820 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:42.418767 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 00:33:42.542935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:42.542830 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:42.615992 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:42.547143 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:43.085943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:42.713529 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 30 00:33:43.110936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:42.714551 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:33:43.110936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:43.060855 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:33:43.110936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:43.086564 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" podUID="44355452-926f-4608-b0eb-24bb7e30dcbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:43.110936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:43.084929 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:33:43.110936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:43.087092 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" podUID="27699192-a6f9-4c03-a63d-e19a122dff3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:43.156033 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:33:43.119402 5001 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.962s" Oct 30 00:33:43.705179 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:43.705149 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-415d24-8cc6f76cf-vnt5d"] Oct 30 00:33:43.952849 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:43.134370 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:33:43.953758 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:43.432452 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:43.954035 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:43.551987 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:43.954276 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:43.649123 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 00:33:44.027046 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:43.338386 5001 patch_prober.go:29] interesting pod/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:33:44.027382 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.027341 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" podUID="b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:44.106390 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.106342 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 30 00:33:44.114796 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.114752 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:44.136815 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.117107 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:44.138595 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.117662 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:33:44.203715 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.189261 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:33:44.271360 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.271229 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" podUID="44355452-926f-4608-b0eb-24bb7e30dcbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:44.272124 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.203197 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-415d24-8cc6f76cf-vnt5d"] Oct 30 00:33:44.273242 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.203334 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-415d24-8cc6f76cf-vnt5d" Oct 30 00:33:44.284110 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.266891 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnpss\" (UniqueName: \"kubernetes.io/projected/be687896-3ef5-4bfc-b170-3e91fcc13a3e-kube-api-access-vnpss\") pod \"payload-pre-processing-e2e-shared-b-w0-415d24-8cc6f76cf-vnt5d\" (UID: \"be687896-3ef5-4bfc-b170-3e91fcc13a3e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-415d24-8cc6f76cf-vnt5d" Oct 30 00:33:44.284721 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.284686 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be687896-3ef5-4bfc-b170-3e91fcc13a3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-415d24-8cc6f76cf-vnt5d\" (UID: \"be687896-3ef5-4bfc-b170-3e91fcc13a3e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-415d24-8cc6f76cf-vnt5d" Oct 30 00:33:44.317450 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.317420 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-415d24\"" Oct 30 00:33:44.317796 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.317762 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-415d24-dockercfg-98qxk\"" Oct 30 00:33:44.405417 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.405345 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7584-61fb81" podStartSLOduration=18.40532281 podStartE2EDuration="18.40532281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:39.969570274 +0000 UTC m=+1735.175071456" watchObservedRunningTime="2026-10-30 00:33:44.40532281 +0000 UTC m=+1739.610823992" Oct 30 00:33:44.467392 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.428699 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:33:44.467703 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.467681 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:33:44.468035 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.468016 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-415d24-9fd8c87c8-wn6vg"] Oct 30 00:33:44.469650 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.464327 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be687896-3ef5-4bfc-b170-3e91fcc13a3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-415d24-8cc6f76cf-vnt5d\" (UID: \"be687896-3ef5-4bfc-b170-3e91fcc13a3e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-415d24-8cc6f76cf-vnt5d" Oct 30 00:33:44.522962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.432808 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be687896-3ef5-4bfc-b170-3e91fcc13a3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-415d24-8cc6f76cf-vnt5d\" (UID: \"be687896-3ef5-4bfc-b170-3e91fcc13a3e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-415d24-8cc6f76cf-vnt5d" Oct 30 00:33:44.600951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.595815 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnpss\" (UniqueName: \"kubernetes.io/projected/be687896-3ef5-4bfc-b170-3e91fcc13a3e-kube-api-access-vnpss\") pod \"payload-pre-processing-e2e-shared-b-w0-415d24-8cc6f76cf-vnt5d\" (UID: \"be687896-3ef5-4bfc-b170-3e91fcc13a3e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-415d24-8cc6f76cf-vnt5d" Oct 30 00:33:44.675947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.675754 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-415d24-9fd8c87c8-wn6vg"] Oct 30 00:33:44.675947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.675841 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:33:44.675947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.675864 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:33:44.675947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.675903 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:33:44.676295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.675961 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:33:44.676295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.676051 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-415d24-9fd8c87c8-wn6vg" Oct 30 00:33:44.688614 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.688585 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnpss\" (UniqueName: \"kubernetes.io/projected/be687896-3ef5-4bfc-b170-3e91fcc13a3e-kube-api-access-vnpss\") pod \"payload-pre-processing-e2e-shared-b-w0-415d24-8cc6f76cf-vnt5d\" (UID: \"be687896-3ef5-4bfc-b170-3e91fcc13a3e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-415d24-8cc6f76cf-vnt5d" Oct 30 00:33:44.694983 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.692222 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ecb34a-7ffdbf96c-8lgp2"] Oct 30 00:33:44.695942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.695155 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:33:44.704224 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.700405 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:33:44.822166 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.819743 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hnkf\" (UniqueName: \"kubernetes.io/projected/3f3b775e-8102-4e77-a2f7-272ac5d6ad5a-kube-api-access-9hnkf\") pod \"payload-processing-e2e-shared-b-w0-415d24-9fd8c87c8-wn6vg\" (UID: \"3f3b775e-8102-4e77-a2f7-272ac5d6ad5a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-415d24-9fd8c87c8-wn6vg" Oct 30 00:33:44.822166 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.820232 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-415d24-8cc6f76cf-vnt5d" Oct 30 00:33:44.822166 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.820385 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f3b775e-8102-4e77-a2f7-272ac5d6ad5a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-415d24-9fd8c87c8-wn6vg\" (UID: \"3f3b775e-8102-4e77-a2f7-272ac5d6ad5a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-415d24-9fd8c87c8-wn6vg" Oct 30 00:33:44.830620 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.830539 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-ecb34a-7c4f79cf86-6l8nz"] Oct 30 00:33:44.830794 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.830685 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" podStartSLOduration=12.830665001 podStartE2EDuration="12.830665001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:44.145405139 +0000 UTC m=+1739.350906343" watchObservedRunningTime="2026-10-30 00:33:44.830665001 +0000 UTC m=+1740.036166182" Oct 30 00:33:44.884950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.883143 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:33:44.888152 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.887602 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 30 00:33:44.888152 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.887653 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" podUID="1222e77c-6ae5-4ce1-a15f-caa491260730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 30 00:33:44.942434 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.917841 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:33:45.102216 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.970988 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:45.102216 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.977716 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f3b775e-8102-4e77-a2f7-272ac5d6ad5a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-415d24-9fd8c87c8-wn6vg\" (UID: \"3f3b775e-8102-4e77-a2f7-272ac5d6ad5a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-415d24-9fd8c87c8-wn6vg" Oct 30 00:33:45.102216 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:44.977806 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hnkf\" (UniqueName: \"kubernetes.io/projected/3f3b775e-8102-4e77-a2f7-272ac5d6ad5a-kube-api-access-9hnkf\") pod \"payload-processing-e2e-shared-b-w0-415d24-9fd8c87c8-wn6vg\" (UID: \"3f3b775e-8102-4e77-a2f7-272ac5d6ad5a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-415d24-9fd8c87c8-wn6vg" Oct 30 00:33:45.131993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:45.131959 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hnkf\" (UniqueName: \"kubernetes.io/projected/3f3b775e-8102-4e77-a2f7-272ac5d6ad5a-kube-api-access-9hnkf\") pod \"payload-processing-e2e-shared-b-w0-415d24-9fd8c87c8-wn6vg\" (UID: \"3f3b775e-8102-4e77-a2f7-272ac5d6ad5a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-415d24-9fd8c87c8-wn6vg" Oct 30 00:33:45.156857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:45.156822 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f3b775e-8102-4e77-a2f7-272ac5d6ad5a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-415d24-9fd8c87c8-wn6vg\" (UID: \"3f3b775e-8102-4e77-a2f7-272ac5d6ad5a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-415d24-9fd8c87c8-wn6vg" Oct 30 00:33:45.234421 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:45.234389 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:33:45.234709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:45.234694 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:33:45.234841 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:45.234831 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:33:45.234979 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:45.234967 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:33:45.257210 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:45.257179 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:33:45.268142 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:45.268114 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:33:45.396849 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:45.396760 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ecb34a-7c4f79cf86-6l8nz" event={"ID":"0261228f-3733-4993-893e-43c742c7bd5e","Type":"ContainerStarted","Data":"25e48c2a3fc075d6c0f95cbed64bb6e6636e5d00bc2635b77952d4f7b5ee6711"} Oct 30 00:33:45.402838 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:45.402562 5001 generic.go:356] "Generic (PLEG): container finished" podID="89e9208c-ef2c-483d-a9bb-666416fd9076" containerID="845d25c8a2445cf8c863fde6a4eee6d96a356c356e4ef6c2b93321f82614c778" exitCode=0 Oct 30 00:33:45.402838 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:45.402651 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1c718-mjsnx" event={"ID":"89e9208c-ef2c-483d-a9bb-666416fd9076","Type":"ContainerDied","Data":"845d25c8a2445cf8c863fde6a4eee6d96a356c356e4ef6c2b93321f82614c778"} Oct 30 00:33:45.409988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:45.406257 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-415d24-9fd8c87c8-wn6vg" Oct 30 00:33:45.428443 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:45.428394 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" event={"ID":"1222e77c-6ae5-4ce1-a15f-caa491260730","Type":"ContainerStarted","Data":"6cc703f7001268a7b52c39bea2e3c7a0fbd6c65f5455116e64b37e03d1346ee3"} Oct 30 00:33:45.504213 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:45.501860 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ecb34a-7ffdbf96c-8lgp2" event={"ID":"aca30d4b-8350-4bfd-ba90-54e70dacf765","Type":"ContainerStarted","Data":"2d9c1cf0fefda535b52b87921c9e3752803c6e1de1a90bc46be95d007cd32f38"} Oct 30 00:33:45.504213 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:45.501956 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ecb34a-7ffdbf96c-8lgp2" event={"ID":"aca30d4b-8350-4bfd-ba90-54e70dacf765","Type":"ContainerStarted","Data":"f5c1bc9ee34f9a08811dbd6279cde733d1e43ec914fea07bf9e7f9fa962bd8ab"} Oct 30 00:33:45.833934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:45.833837 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 30 00:33:45.833934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:45.833897 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" podUID="1222e77c-6ae5-4ce1-a15f-caa491260730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 30 00:33:46.214424 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.214392 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7584-61fb81"] Oct 30 00:33:46.223951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.215051 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7584-61fb81" podUID="9b197a22-4dd0-4d03-8379-488fa2ad6a5e" containerName="test-curl-7584-61fb81" containerID="cri-o://38bc6e344ce6fe603763cdfaf2ae897744f809e4de943fdf630028be974bede6" gracePeriod=2 Oct 30 00:33:46.224678 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.222679 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ecb34a-7ffdbf96c-8lgp2" podStartSLOduration=11.222663975 podStartE2EDuration="11.222663975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:45.975936488 +0000 UTC m=+1741.181437696" watchObservedRunningTime="2026-10-30 00:33:46.222663975 +0000 UTC m=+1741.428165158" Oct 30 00:33:46.236552 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.222734 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7584-61fb81"] Oct 30 00:33:46.238335 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:33:46.238289 5001 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7584-61fb81\" is forbidden: User \"system:node:ip-10-0-143-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-120.ec2.internal' and this object" podUID="9b197a22-4dd0-4d03-8379-488fa2ad6a5e" pod="opendatahub/test-curl-7584-61fb81" Oct 30 00:33:46.291975 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.291946 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-415d24-8cc6f76cf-vnt5d"] Oct 30 00:33:46.301459 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:33:46.295199 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe687896_3ef5_4bfc_b170_3e91fcc13a3e.slice/crio-01a48f11bc6804fc88fa9ac4a9f7b710e63de556d1ad2bc26d3b484c98202b7e WatchSource:0}: Error finding container 01a48f11bc6804fc88fa9ac4a9f7b710e63de556d1ad2bc26d3b484c98202b7e: Status 404 returned error can't find the container with id 01a48f11bc6804fc88fa9ac4a9f7b710e63de556d1ad2bc26d3b484c98202b7e Oct 30 00:33:46.302361 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:33:46.302328 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f3b775e_8102_4e77_a2f7_272ac5d6ad5a.slice/crio-074b5448875a1fbd36d5ac7e128da5c6fe5a80f737d583be7bcb8d5cb4d50fae WatchSource:0}: Error finding container 074b5448875a1fbd36d5ac7e128da5c6fe5a80f737d583be7bcb8d5cb4d50fae: Status 404 returned error can't find the container with id 074b5448875a1fbd36d5ac7e128da5c6fe5a80f737d583be7bcb8d5cb4d50fae Oct 30 00:33:46.305286 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.305259 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-415d24-9fd8c87c8-wn6vg"] Oct 30 00:33:46.327531 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.327501 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-05c6b2-55b55d747-dgrp9"] Oct 30 00:33:46.328770 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.328747 5001 cpu_manager.go:414] "containerMap: removing container" podUID="9b197a22-4dd0-4d03-8379-488fa2ad6a5e" containerName="test-curl-7584-61fb81" Oct 30 00:33:46.328982 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.328963 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b197a22-4dd0-4d03-8379-488fa2ad6a5e" containerName="test-curl-7584-61fb81" Oct 30 00:33:46.329359 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.329341 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b197a22-4dd0-4d03-8379-488fa2ad6a5e" containerName="test-curl-7584-61fb81" Oct 30 00:33:46.342619 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.341611 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-05c6b2-55b55d747-dgrp9" Oct 30 00:33:46.353703 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:33:46.353544 5001 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7584-61fb81\" is forbidden: User \"system:node:ip-10-0-143-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-120.ec2.internal' and this object" podUID="9b197a22-4dd0-4d03-8379-488fa2ad6a5e" pod="opendatahub/test-curl-7584-61fb81" Oct 30 00:33:46.397878 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.383061 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-05c6b2-dockercfg-mjgbj\"" Oct 30 00:33:46.425053 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.423766 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-05c6b2\"" Oct 30 00:33:46.687275 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.687210 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwjqw\" (UniqueName: \"kubernetes.io/projected/bd4c0988-ad1a-4288-85f9-573a02594fe5-kube-api-access-wwjqw\") pod \"payload-pre-processing-e2e-shared-b-w2-05c6b2-55b55d747-dgrp9\" (UID: \"bd4c0988-ad1a-4288-85f9-573a02594fe5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-05c6b2-55b55d747-dgrp9" Oct 30 00:33:46.722944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.714624 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd4c0988-ad1a-4288-85f9-573a02594fe5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-05c6b2-55b55d747-dgrp9\" (UID: \"bd4c0988-ad1a-4288-85f9-573a02594fe5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-05c6b2-55b55d747-dgrp9" Oct 30 00:33:46.842020 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.836949 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 30 00:33:46.842020 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.837009 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" podUID="1222e77c-6ae5-4ce1-a15f-caa491260730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 30 00:33:46.872362 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.871966 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-05c6b2-55b55d747-dgrp9"] Oct 30 00:33:46.905532 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.888312 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-415d24-8cc6f76cf-vnt5d" event={"ID":"be687896-3ef5-4bfc-b170-3e91fcc13a3e","Type":"ContainerStarted","Data":"01a48f11bc6804fc88fa9ac4a9f7b710e63de556d1ad2bc26d3b484c98202b7e"} Oct 30 00:33:46.905532 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.892112 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ecb34a-7c4f79cf86-6l8nz" event={"ID":"0261228f-3733-4993-893e-43c742c7bd5e","Type":"ContainerStarted","Data":"f333c84868e04ee11a4d80dc0251fa3008a4c0ca379f0ce0a97a7166ba0a9e2a"} Oct 30 00:33:46.905532 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.893654 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ecb34a-7c4f79cf86-6l8nz" Oct 30 00:33:46.905532 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.896784 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-415d24-9fd8c87c8-wn6vg" event={"ID":"3f3b775e-8102-4e77-a2f7-272ac5d6ad5a","Type":"ContainerStarted","Data":"074b5448875a1fbd36d5ac7e128da5c6fe5a80f737d583be7bcb8d5cb4d50fae"} Oct 30 00:33:46.905532 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.897770 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ecb34a-7ffdbf96c-8lgp2" Oct 30 00:33:46.905532 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.904197 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-564c8f-wgm5x"] Oct 30 00:33:46.943111 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.920820 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwjqw\" (UniqueName: \"kubernetes.io/projected/bd4c0988-ad1a-4288-85f9-573a02594fe5-kube-api-access-wwjqw\") pod \"payload-pre-processing-e2e-shared-b-w2-05c6b2-55b55d747-dgrp9\" (UID: \"bd4c0988-ad1a-4288-85f9-573a02594fe5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-05c6b2-55b55d747-dgrp9" Oct 30 00:33:46.943111 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.921970 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd4c0988-ad1a-4288-85f9-573a02594fe5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-05c6b2-55b55d747-dgrp9\" (UID: \"bd4c0988-ad1a-4288-85f9-573a02594fe5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-05c6b2-55b55d747-dgrp9" Oct 30 00:33:46.943111 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.925817 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd4c0988-ad1a-4288-85f9-573a02594fe5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-05c6b2-55b55d747-dgrp9\" (UID: \"bd4c0988-ad1a-4288-85f9-573a02594fe5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-05c6b2-55b55d747-dgrp9" Oct 30 00:33:46.944905 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.944879 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-564c8f-wgm5x" Oct 30 00:33:46.948502 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.948472 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwjqw\" (UniqueName: \"kubernetes.io/projected/bd4c0988-ad1a-4288-85f9-573a02594fe5-kube-api-access-wwjqw\") pod \"payload-pre-processing-e2e-shared-b-w2-05c6b2-55b55d747-dgrp9\" (UID: \"bd4c0988-ad1a-4288-85f9-573a02594fe5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-05c6b2-55b55d747-dgrp9" Oct 30 00:33:46.952454 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:33:46.952425 5001 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7584-61fb81\" is forbidden: User \"system:node:ip-10-0-143-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-120.ec2.internal' and this object" podUID="9b197a22-4dd0-4d03-8379-488fa2ad6a5e" pod="opendatahub/test-curl-7584-61fb81" Oct 30 00:33:46.957985 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.957963 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-05c6b2-6f8558fd8b-kslr7"] Oct 30 00:33:46.963040 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.963020 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-05c6b2-6f8558fd8b-kslr7" Oct 30 00:33:46.966761 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.966699 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-564c8f-wgm5x"] Oct 30 00:33:46.976998 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.976977 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-05c6b2-6f8558fd8b-kslr7"] Oct 30 00:33:46.978007 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.977957 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ecb34a-7c4f79cf86-6l8nz" podStartSLOduration=10.977940333 podStartE2EDuration="10.977940333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:46.941452906 +0000 UTC m=+1742.146954088" watchObservedRunningTime="2026-10-30 00:33:46.977940333 +0000 UTC m=+1742.183441516" Oct 30 00:33:46.991333 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:46.991207 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-05c6b2-55b55d747-dgrp9" Oct 30 00:33:46.996178 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:33:46.996054 5001 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7584-61fb81\" is forbidden: User \"system:node:ip-10-0-143-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-120.ec2.internal' and this object" podUID="9b197a22-4dd0-4d03-8379-488fa2ad6a5e" pod="opendatahub/test-curl-7584-61fb81" Oct 30 00:33:47.200262 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:47.169555 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63d72c52-189d-4842-8946-d073e53b6879-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-05c6b2-6f8558fd8b-kslr7\" (UID: \"63d72c52-189d-4842-8946-d073e53b6879\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-05c6b2-6f8558fd8b-kslr7" Oct 30 00:33:47.200262 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:47.169745 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjcbr\" (UniqueName: \"kubernetes.io/projected/cf200711-0050-4fae-8fe0-385b6e39dbe2-kube-api-access-tjcbr\") pod \"maas-api-revoke-keys-e2e-worker-w3-564c8f-wgm5x\" (UID: \"cf200711-0050-4fae-8fe0-385b6e39dbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-564c8f-wgm5x" Oct 30 00:33:47.200262 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:47.169968 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-424mg\" (UniqueName: \"kubernetes.io/projected/63d72c52-189d-4842-8946-d073e53b6879-kube-api-access-424mg\") pod \"payload-processing-e2e-shared-b-w2-05c6b2-6f8558fd8b-kslr7\" (UID: \"63d72c52-189d-4842-8946-d073e53b6879\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-05c6b2-6f8558fd8b-kslr7" Oct 30 00:33:47.200262 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:47.169996 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf200711-0050-4fae-8fe0-385b6e39dbe2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-564c8f-wgm5x\" (UID: \"cf200711-0050-4fae-8fe0-385b6e39dbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-564c8f-wgm5x" Oct 30 00:33:47.200262 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:47.170021 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf200711-0050-4fae-8fe0-385b6e39dbe2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-564c8f-wgm5x\" (UID: \"cf200711-0050-4fae-8fe0-385b6e39dbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-564c8f-wgm5x" Oct 30 00:33:47.272182 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:47.272152 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-424mg\" (UniqueName: \"kubernetes.io/projected/63d72c52-189d-4842-8946-d073e53b6879-kube-api-access-424mg\") pod \"payload-processing-e2e-shared-b-w2-05c6b2-6f8558fd8b-kslr7\" (UID: \"63d72c52-189d-4842-8946-d073e53b6879\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-05c6b2-6f8558fd8b-kslr7" Oct 30 00:33:47.272364 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:47.272194 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf200711-0050-4fae-8fe0-385b6e39dbe2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-564c8f-wgm5x\" (UID: \"cf200711-0050-4fae-8fe0-385b6e39dbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-564c8f-wgm5x" Oct 30 00:33:47.272364 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:47.272220 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf200711-0050-4fae-8fe0-385b6e39dbe2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-564c8f-wgm5x\" (UID: \"cf200711-0050-4fae-8fe0-385b6e39dbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-564c8f-wgm5x" Oct 30 00:33:47.272516 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:47.272376 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63d72c52-189d-4842-8946-d073e53b6879-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-05c6b2-6f8558fd8b-kslr7\" (UID: \"63d72c52-189d-4842-8946-d073e53b6879\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-05c6b2-6f8558fd8b-kslr7" Oct 30 00:33:47.272581 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:47.272538 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjcbr\" (UniqueName: \"kubernetes.io/projected/cf200711-0050-4fae-8fe0-385b6e39dbe2-kube-api-access-tjcbr\") pod \"maas-api-revoke-keys-e2e-worker-w3-564c8f-wgm5x\" (UID: \"cf200711-0050-4fae-8fe0-385b6e39dbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-564c8f-wgm5x" Oct 30 00:33:47.273609 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:47.273581 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf200711-0050-4fae-8fe0-385b6e39dbe2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-564c8f-wgm5x\" (UID: \"cf200711-0050-4fae-8fe0-385b6e39dbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-564c8f-wgm5x" Oct 30 00:33:47.274548 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:47.274516 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63d72c52-189d-4842-8946-d073e53b6879-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-05c6b2-6f8558fd8b-kslr7\" (UID: \"63d72c52-189d-4842-8946-d073e53b6879\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-05c6b2-6f8558fd8b-kslr7" Oct 30 00:33:47.281740 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:47.279454 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf200711-0050-4fae-8fe0-385b6e39dbe2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-564c8f-wgm5x\" (UID: \"cf200711-0050-4fae-8fe0-385b6e39dbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-564c8f-wgm5x" Oct 30 00:33:47.295622 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:47.295559 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjcbr\" (UniqueName: \"kubernetes.io/projected/cf200711-0050-4fae-8fe0-385b6e39dbe2-kube-api-access-tjcbr\") pod \"maas-api-revoke-keys-e2e-worker-w3-564c8f-wgm5x\" (UID: \"cf200711-0050-4fae-8fe0-385b6e39dbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-564c8f-wgm5x" Oct 30 00:33:47.297529 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:47.297462 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-424mg\" (UniqueName: \"kubernetes.io/projected/63d72c52-189d-4842-8946-d073e53b6879-kube-api-access-424mg\") pod \"payload-processing-e2e-shared-b-w2-05c6b2-6f8558fd8b-kslr7\" (UID: \"63d72c52-189d-4842-8946-d073e53b6879\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-05c6b2-6f8558fd8b-kslr7" Oct 30 00:33:47.397118 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:33:47.397080 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:33:47.399253 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:47.399226 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1c718-mjsnx" Oct 30 00:33:47.548642 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:47.548606 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-564c8f-wgm5x" Oct 30 00:33:47.554593 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:47.554005 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-05c6b2-6f8558fd8b-kslr7" Oct 30 00:33:47.587087 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:47.585688 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89e9208c-ef2c-483d-a9bb-666416fd9076-kube-api-access-kc4vj\" (UniqueName: \"kubernetes.io/projected/89e9208c-ef2c-483d-a9bb-666416fd9076-kube-api-access-kc4vj\") pod \"89e9208c-ef2c-483d-a9bb-666416fd9076\" (UID: \"89e9208c-ef2c-483d-a9bb-666416fd9076\") " Oct 30 00:33:47.587087 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:47.585799 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89e9208c-ef2c-483d-a9bb-666416fd9076-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e9208c-ef2c-483d-a9bb-666416fd9076-maas-api-openshift-ca\") pod \"89e9208c-ef2c-483d-a9bb-666416fd9076\" (UID: \"89e9208c-ef2c-483d-a9bb-666416fd9076\") " Oct 30 00:33:47.587087 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:47.585865 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89e9208c-ef2c-483d-a9bb-666416fd9076-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e9208c-ef2c-483d-a9bb-666416fd9076-maas-api-xks-ca\") pod \"89e9208c-ef2c-483d-a9bb-666416fd9076\" (UID: \"89e9208c-ef2c-483d-a9bb-666416fd9076\") " Oct 30 00:33:47.589665 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:47.589637 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89e9208c-ef2c-483d-a9bb-666416fd9076-maas-api-openshift-ca" pod "89e9208c-ef2c-483d-a9bb-666416fd9076" (UID: "89e9208c-ef2c-483d-a9bb-666416fd9076"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:33:47.591833 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:47.591805 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89e9208c-ef2c-483d-a9bb-666416fd9076-maas-api-xks-ca" pod "89e9208c-ef2c-483d-a9bb-666416fd9076" (UID: "89e9208c-ef2c-483d-a9bb-666416fd9076"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:33:47.591981 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:47.591813 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89e9208c-ef2c-483d-a9bb-666416fd9076-kube-api-access-kc4vj" pod "89e9208c-ef2c-483d-a9bb-666416fd9076" (UID: "89e9208c-ef2c-483d-a9bb-666416fd9076"). InnerVolumeSpecName "kube-api-access-kc4vj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:33:47.602782 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:47.602755 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-05c6b2-55b55d747-dgrp9"] Oct 30 00:33:47.603044 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:33:47.603019 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd4c0988_ad1a_4288_85f9_573a02594fe5.slice/crio-0b236e3b4ead1bd1351158535dbe817a94668d6adb6f132268e0d5bcc714ed7b WatchSource:0}: Error finding container 0b236e3b4ead1bd1351158535dbe817a94668d6adb6f132268e0d5bcc714ed7b: Status 404 returned error can't find the container with id 0b236e3b4ead1bd1351158535dbe817a94668d6adb6f132268e0d5bcc714ed7b Oct 30 00:33:47.688966 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:47.688927 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc4vj\" (UniqueName: \"kubernetes.io/projected/89e9208c-ef2c-483d-a9bb-666416fd9076-kube-api-access-kc4vj\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:33:47.688966 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:47.688962 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e9208c-ef2c-483d-a9bb-666416fd9076-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:33:47.689200 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:47.688978 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e9208c-ef2c-483d-a9bb-666416fd9076-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:33:47.869888 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:47.869782 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 30 00:33:47.869888 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:47.869842 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" podUID="1222e77c-6ae5-4ce1-a15f-caa491260730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 30 00:33:47.992458 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:47.987203 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-564c8f-wgm5x"] Oct 30 00:33:48.051317 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:33:48.036740 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf200711_0050_4fae_8fe0_385b6e39dbe2.slice/crio-2fa7b2788d53b88158f6f41c86c0dd1a3c2d96dc9975765fe2d141ea4d563741 WatchSource:0}: Error finding container 2fa7b2788d53b88158f6f41c86c0dd1a3c2d96dc9975765fe2d141ea4d563741: Status 404 returned error can't find the container with id 2fa7b2788d53b88158f6f41c86c0dd1a3c2d96dc9975765fe2d141ea4d563741 Oct 30 00:33:48.157062 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:48.156897 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-05c6b2-55b55d747-dgrp9" event={"ID":"bd4c0988-ad1a-4288-85f9-573a02594fe5","Type":"ContainerStarted","Data":"0714c29b98a703ee902d129d3a2aae10482bf8d17f7a517136cc518206f53e8c"} Oct 30 00:33:48.157062 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:48.156977 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-05c6b2-55b55d747-dgrp9" event={"ID":"bd4c0988-ad1a-4288-85f9-573a02594fe5","Type":"ContainerStarted","Data":"0b236e3b4ead1bd1351158535dbe817a94668d6adb6f132268e0d5bcc714ed7b"} Oct 30 00:33:48.157062 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:48.157028 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-05c6b2-55b55d747-dgrp9" Oct 30 00:33:48.159632 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:48.159606 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-05c6b2-6f8558fd8b-kslr7"] Oct 30 00:33:48.161283 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:33:48.161249 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d72c52_189d_4842_8946_d073e53b6879.slice/crio-afaaa7e24d7f4639b50492e2b3e6eaa5041154ea9d31f0abcc00b4ee97585c1a WatchSource:0}: Error finding container afaaa7e24d7f4639b50492e2b3e6eaa5041154ea9d31f0abcc00b4ee97585c1a: Status 404 returned error can't find the container with id afaaa7e24d7f4639b50492e2b3e6eaa5041154ea9d31f0abcc00b4ee97585c1a Oct 30 00:33:48.163320 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:48.162446 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-415d24-9fd8c87c8-wn6vg" event={"ID":"3f3b775e-8102-4e77-a2f7-272ac5d6ad5a","Type":"ContainerStarted","Data":"5910f3d6ff3668603ced6ac7197ee3fad977de116c28dbcc9f2fc1a6d5f13dce"} Oct 30 00:33:48.163320 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:48.163177 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-415d24-9fd8c87c8-wn6vg" Oct 30 00:33:48.167848 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:48.167821 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-415d24-8cc6f76cf-vnt5d" event={"ID":"be687896-3ef5-4bfc-b170-3e91fcc13a3e","Type":"ContainerStarted","Data":"84298f705f2b88ee0b66b590765433bfb83e3b2f4996c5e0600fa5a98f2ff830"} Oct 30 00:33:48.168132 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:48.167967 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-415d24-8cc6f76cf-vnt5d" Oct 30 00:33:48.171554 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:48.171534 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1c718-mjsnx" Oct 30 00:33:48.181175 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:48.181144 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1c718-mjsnx" event={"ID":"89e9208c-ef2c-483d-a9bb-666416fd9076","Type":"ContainerDied","Data":"ef30773f220627929e62093cf7836d2e11532fb5ca0152d8df86c61cf2569085"} Oct 30 00:33:48.181270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:48.181202 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef30773f220627929e62093cf7836d2e11532fb5ca0152d8df86c61cf2569085" Oct 30 00:33:48.193398 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:48.193348 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-05c6b2-55b55d747-dgrp9" podStartSLOduration=2.193332928 podStartE2EDuration="2.193332928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:48.191805775 +0000 UTC m=+1743.397306958" watchObservedRunningTime="2026-10-30 00:33:48.193332928 +0000 UTC m=+1743.398834113" Oct 30 00:33:48.232094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:48.232037 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-415d24-8cc6f76cf-vnt5d" podStartSLOduration=9.23201629 podStartE2EDuration="9.23201629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:48.228904304 +0000 UTC m=+1743.434405488" watchObservedRunningTime="2026-10-30 00:33:48.23201629 +0000 UTC m=+1743.437517488" Oct 30 00:33:48.266628 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:48.266033 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-415d24-9fd8c87c8-wn6vg" podStartSLOduration=9.266013104 podStartE2EDuration="9.266013104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:48.264570667 +0000 UTC m=+1743.470071851" watchObservedRunningTime="2026-10-30 00:33:48.266013104 +0000 UTC m=+1743.471514285" Oct 30 00:33:48.750550 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:48.750517 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7584-61fb81" Oct 30 00:33:48.833309 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:48.833267 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 30 00:33:48.833475 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:48.833328 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" podUID="1222e77c-6ae5-4ce1-a15f-caa491260730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 30 00:33:48.868431 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:48.868062 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b197a22-4dd0-4d03-8379-488fa2ad6a5e-kube-api-access-t4cxl\" (UniqueName: \"kubernetes.io/projected/9b197a22-4dd0-4d03-8379-488fa2ad6a5e-kube-api-access-t4cxl\") pod \"9b197a22-4dd0-4d03-8379-488fa2ad6a5e\" (UID: \"9b197a22-4dd0-4d03-8379-488fa2ad6a5e\") " Oct 30 00:33:48.870800 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:48.870770 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b197a22-4dd0-4d03-8379-488fa2ad6a5e-kube-api-access-t4cxl" pod "9b197a22-4dd0-4d03-8379-488fa2ad6a5e" (UID: "9b197a22-4dd0-4d03-8379-488fa2ad6a5e"). InnerVolumeSpecName "kube-api-access-t4cxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:33:48.945757 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:48.945721 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b197a22-4dd0-4d03-8379-488fa2ad6a5e" path="/var/lib/kubelet/pods/9b197a22-4dd0-4d03-8379-488fa2ad6a5e/volumes" Oct 30 00:33:48.970975 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:48.970948 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4cxl\" (UniqueName: \"kubernetes.io/projected/9b197a22-4dd0-4d03-8379-488fa2ad6a5e-kube-api-access-t4cxl\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:33:49.187748 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:49.187706 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-05c6b2-6f8558fd8b-kslr7" event={"ID":"63d72c52-189d-4842-8946-d073e53b6879","Type":"ContainerStarted","Data":"30937ffaa0502579eb4c311ae9511628d86d5ca3e656d84ff364d87e32d1443d"} Oct 30 00:33:49.187959 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:49.187773 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-05c6b2-6f8558fd8b-kslr7" event={"ID":"63d72c52-189d-4842-8946-d073e53b6879","Type":"ContainerStarted","Data":"afaaa7e24d7f4639b50492e2b3e6eaa5041154ea9d31f0abcc00b4ee97585c1a"} Oct 30 00:33:49.188695 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:49.188668 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-05c6b2-6f8558fd8b-kslr7" Oct 30 00:33:49.190223 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:49.190192 5001 generic.go:356] "Generic (PLEG): container finished" podID="9b197a22-4dd0-4d03-8379-488fa2ad6a5e" containerID="38bc6e344ce6fe603763cdfaf2ae897744f809e4de943fdf630028be974bede6" exitCode=137 Oct 30 00:33:49.190333 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:49.190223 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7584-61fb81" Oct 30 00:33:49.190333 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:49.190283 5001 scope.go:122] "RemoveContainer" containerID="38bc6e344ce6fe603763cdfaf2ae897744f809e4de943fdf630028be974bede6" Oct 30 00:33:49.194624 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:49.194214 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-564c8f-wgm5x" event={"ID":"cf200711-0050-4fae-8fe0-385b6e39dbe2","Type":"ContainerStarted","Data":"7122efbb1cf2e6f43e62091d7b67135731b5975c9b534f5a43cd0d76fa6cabb6"} Oct 30 00:33:49.194624 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:49.194259 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-564c8f-wgm5x" event={"ID":"cf200711-0050-4fae-8fe0-385b6e39dbe2","Type":"ContainerStarted","Data":"2fa7b2788d53b88158f6f41c86c0dd1a3c2d96dc9975765fe2d141ea4d563741"} Oct 30 00:33:49.229605 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:49.228760 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-05c6b2-6f8558fd8b-kslr7" podStartSLOduration=3.228740346 podStartE2EDuration="3.228740346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:49.225312987 +0000 UTC m=+1744.430814169" watchObservedRunningTime="2026-10-30 00:33:49.228740346 +0000 UTC m=+1744.434241527" Oct 30 00:33:49.236184 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:49.235858 5001 scope.go:122] "RemoveContainer" containerID="38bc6e344ce6fe603763cdfaf2ae897744f809e4de943fdf630028be974bede6" Oct 30 00:33:49.236554 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:33:49.236455 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38bc6e344ce6fe603763cdfaf2ae897744f809e4de943fdf630028be974bede6\": container with ID starting with 38bc6e344ce6fe603763cdfaf2ae897744f809e4de943fdf630028be974bede6 not found: ID does not exist" containerID="38bc6e344ce6fe603763cdfaf2ae897744f809e4de943fdf630028be974bede6" Oct 30 00:33:49.236554 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:49.236494 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38bc6e344ce6fe603763cdfaf2ae897744f809e4de943fdf630028be974bede6"} err="failed to get container status \"38bc6e344ce6fe603763cdfaf2ae897744f809e4de943fdf630028be974bede6\": rpc error: code = NotFound desc = could not find container \"38bc6e344ce6fe603763cdfaf2ae897744f809e4de943fdf630028be974bede6\": container with ID starting with 38bc6e344ce6fe603763cdfaf2ae897744f809e4de943fdf630028be974bede6 not found: ID does not exist" Oct 30 00:33:49.265007 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:49.264943 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-564c8f-wgm5x" podStartSLOduration=3.264927311 podStartE2EDuration="3.264927311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:49.259875356 +0000 UTC m=+1744.465376564" watchObservedRunningTime="2026-10-30 00:33:49.264927311 +0000 UTC m=+1744.470428530" Oct 30 00:33:49.799186 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:49.799101 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:33:49.799186 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:49.799174 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" podUID="ff9c1b8a-7343-4566-9611-7dc87cb6fea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:49.833348 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:49.833231 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 30 00:33:49.833348 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:49.833292 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" podUID="1222e77c-6ae5-4ce1-a15f-caa491260730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 30 00:33:50.241984 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:50.235131 5001 generic.go:356] "Generic (PLEG): container finished" podID="cf200711-0050-4fae-8fe0-385b6e39dbe2" containerID="7122efbb1cf2e6f43e62091d7b67135731b5975c9b534f5a43cd0d76fa6cabb6" exitCode=0 Oct 30 00:33:50.241984 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:50.240527 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-564c8f-wgm5x" event={"ID":"cf200711-0050-4fae-8fe0-385b6e39dbe2","Type":"ContainerDied","Data":"7122efbb1cf2e6f43e62091d7b67135731b5975c9b534f5a43cd0d76fa6cabb6"} Oct 30 00:33:50.850346 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:50.849601 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:33:50.850346 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:50.850020 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 30 00:33:50.850346 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:50.850057 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" podUID="1222e77c-6ae5-4ce1-a15f-caa491260730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 30 00:33:50.874016 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:50.873964 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" containerID="cri-o://03bebce622ce78c29cf07a2e18e5c4ff70786e27d839711ea040730c3ca8fd8b" gracePeriod=45 Oct 30 00:33:51.062098 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.058690 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh"] Oct 30 00:33:51.077255 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.072905 5001 cpu_manager.go:414] "containerMap: removing container" podUID="89e9208c-ef2c-483d-a9bb-666416fd9076" containerName="revoke-keys" Oct 30 00:33:51.077255 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.072956 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89e9208c-ef2c-483d-a9bb-666416fd9076" containerName="revoke-keys" Oct 30 00:33:51.077255 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.073339 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="89e9208c-ef2c-483d-a9bb-666416fd9076" containerName="revoke-keys" Oct 30 00:33:51.099996 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.094228 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:33:51.120681 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.120654 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-f1268c-openshift-default-dockercfg-b2xfh\"" Oct 30 00:33:51.128790 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.128764 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh"] Oct 30 00:33:51.146986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.146954 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn"] Oct 30 00:33:51.265657 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.262393 5001 generic.go:356] "Generic (PLEG): container finished" podID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerID="03bebce622ce78c29cf07a2e18e5c4ff70786e27d839711ea040730c3ca8fd8b" exitCode=0 Oct 30 00:33:51.265657 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.262889 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" event={"ID":"aabed96e-162a-4e58-a3b1-57750140a0e8","Type":"ContainerDied","Data":"03bebce622ce78c29cf07a2e18e5c4ff70786e27d839711ea040730c3ca8fd8b"} Oct 30 00:33:51.265657 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.264631 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-workload-certs\") pod \"e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:33:51.265657 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.264722 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istio-podinfo\") pod \"e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:33:51.265657 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.264815 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7tb2\" (UniqueName: \"kubernetes.io/projected/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-kube-api-access-l7tb2\") pod \"e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:33:51.265657 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.264943 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istio-envoy\") pod \"e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:33:51.265657 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.264973 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istio-token\") pod \"e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:33:51.265657 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.265044 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istio-data\") pod \"e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:33:51.265657 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.265110 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-workload-socket\") pod \"e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:33:51.265657 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.265191 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istiod-ca-cert\") pod \"e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:33:51.265657 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.265218 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-credential-socket\") pod \"e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:33:51.380314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.371708 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-workload-certs\") pod \"e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:33:51.380314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.371836 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istio-podinfo\") pod \"e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:33:51.380314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.371864 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7tb2\" (UniqueName: \"kubernetes.io/projected/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-kube-api-access-l7tb2\") pod \"e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:33:51.380314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.377281 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istio-envoy\") pod \"e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:33:51.380314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.377333 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istio-token\") pod \"e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:33:51.380314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.377439 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istio-data\") pod \"e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:33:51.380314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.377536 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-workload-socket\") pod \"e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:33:51.380314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.377647 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istiod-ca-cert\") pod \"e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:33:51.380314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.377684 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-credential-socket\") pod \"e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:33:51.380314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.378807 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istio-data\") pod \"e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:33:51.380314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.379022 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-workload-socket\") pod \"e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:33:51.380314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.379620 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-credential-socket\") pod \"e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:33:51.380314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.379792 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istiod-ca-cert\") pod \"e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:33:51.380314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.379866 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-workload-certs\") pod \"e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:33:51.386902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.386744 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istio-envoy\") pod \"e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:33:51.408638 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.405058 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istio-podinfo\") pod \"e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:33:51.422022 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.418212 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istio-token\") pod \"e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:33:51.422022 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.421943 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7tb2\" (UniqueName: \"kubernetes.io/projected/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-kube-api-access-l7tb2\") pod \"e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:33:51.422022 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.422308 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:33:51.491115 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.490811 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:33:51.549952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.549502 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-564c8f-wgm5x" Oct 30 00:33:51.592823 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.591103 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aabed96e-162a-4e58-a3b1-57750140a0e8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aabed96e-162a-4e58-a3b1-57750140a0e8-dshm\") pod \"aabed96e-162a-4e58-a3b1-57750140a0e8\" (UID: \"aabed96e-162a-4e58-a3b1-57750140a0e8\") " Oct 30 00:33:51.592823 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.591240 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aabed96e-162a-4e58-a3b1-57750140a0e8-kube-api-access-vlvx5\" (UniqueName: \"kubernetes.io/projected/aabed96e-162a-4e58-a3b1-57750140a0e8-kube-api-access-vlvx5\") pod \"aabed96e-162a-4e58-a3b1-57750140a0e8\" (UID: \"aabed96e-162a-4e58-a3b1-57750140a0e8\") " Oct 30 00:33:51.592823 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.591377 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aabed96e-162a-4e58-a3b1-57750140a0e8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aabed96e-162a-4e58-a3b1-57750140a0e8-model-cache\") pod \"aabed96e-162a-4e58-a3b1-57750140a0e8\" (UID: \"aabed96e-162a-4e58-a3b1-57750140a0e8\") " Oct 30 00:33:51.592823 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.591414 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aabed96e-162a-4e58-a3b1-57750140a0e8-home\" (UniqueName: \"kubernetes.io/empty-dir/aabed96e-162a-4e58-a3b1-57750140a0e8-home\") pod \"aabed96e-162a-4e58-a3b1-57750140a0e8\" (UID: \"aabed96e-162a-4e58-a3b1-57750140a0e8\") " Oct 30 00:33:51.592823 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.591466 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aabed96e-162a-4e58-a3b1-57750140a0e8-tls-certs\" (UniqueName: \"kubernetes.io/secret/aabed96e-162a-4e58-a3b1-57750140a0e8-tls-certs\") pod \"aabed96e-162a-4e58-a3b1-57750140a0e8\" (UID: \"aabed96e-162a-4e58-a3b1-57750140a0e8\") " Oct 30 00:33:51.592823 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.591559 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aabed96e-162a-4e58-a3b1-57750140a0e8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aabed96e-162a-4e58-a3b1-57750140a0e8-tmp-dir\") pod \"aabed96e-162a-4e58-a3b1-57750140a0e8\" (UID: \"aabed96e-162a-4e58-a3b1-57750140a0e8\") " Oct 30 00:33:51.605574 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.605195 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aabed96e-162a-4e58-a3b1-57750140a0e8-home" pod "aabed96e-162a-4e58-a3b1-57750140a0e8" (UID: "aabed96e-162a-4e58-a3b1-57750140a0e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:33:51.638013 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.636179 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aabed96e-162a-4e58-a3b1-57750140a0e8-model-cache" pod "aabed96e-162a-4e58-a3b1-57750140a0e8" (UID: "aabed96e-162a-4e58-a3b1-57750140a0e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:33:51.638013 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.637062 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aabed96e-162a-4e58-a3b1-57750140a0e8-tmp-dir" pod "aabed96e-162a-4e58-a3b1-57750140a0e8" (UID: "aabed96e-162a-4e58-a3b1-57750140a0e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:33:51.697066 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.696745 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf200711-0050-4fae-8fe0-385b6e39dbe2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf200711-0050-4fae-8fe0-385b6e39dbe2-maas-api-openshift-ca\") pod \"cf200711-0050-4fae-8fe0-385b6e39dbe2\" (UID: \"cf200711-0050-4fae-8fe0-385b6e39dbe2\") " Oct 30 00:33:51.697561 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.697425 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf200711-0050-4fae-8fe0-385b6e39dbe2-maas-api-openshift-ca" pod "cf200711-0050-4fae-8fe0-385b6e39dbe2" (UID: "cf200711-0050-4fae-8fe0-385b6e39dbe2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:33:51.697892 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.697873 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf200711-0050-4fae-8fe0-385b6e39dbe2-kube-api-access-tjcbr\" (UniqueName: \"kubernetes.io/projected/cf200711-0050-4fae-8fe0-385b6e39dbe2-kube-api-access-tjcbr\") pod \"cf200711-0050-4fae-8fe0-385b6e39dbe2\" (UID: \"cf200711-0050-4fae-8fe0-385b6e39dbe2\") " Oct 30 00:33:51.698511 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.698492 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf200711-0050-4fae-8fe0-385b6e39dbe2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf200711-0050-4fae-8fe0-385b6e39dbe2-maas-api-xks-ca\") pod \"cf200711-0050-4fae-8fe0-385b6e39dbe2\" (UID: \"cf200711-0050-4fae-8fe0-385b6e39dbe2\") " Oct 30 00:33:51.708034 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.699747 5001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aabed96e-162a-4e58-a3b1-57750140a0e8-tmp-dir\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:33:51.708854 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.708830 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf200711-0050-4fae-8fe0-385b6e39dbe2-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:33:51.709044 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.709027 5001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aabed96e-162a-4e58-a3b1-57750140a0e8-model-cache\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:33:51.709157 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.709140 5001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aabed96e-162a-4e58-a3b1-57750140a0e8-home\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:33:51.739881 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.739822 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aabed96e-162a-4e58-a3b1-57750140a0e8-kube-api-access-vlvx5" pod "aabed96e-162a-4e58-a3b1-57750140a0e8" (UID: "aabed96e-162a-4e58-a3b1-57750140a0e8"). InnerVolumeSpecName "kube-api-access-vlvx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:33:51.794393 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.794045 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:33:51.794393 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.794112 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" podUID="b41b47fa-39f2-4620-b648-797837a6cd1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:33:51.842191 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.841641 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 30 00:33:51.842191 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.841703 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" podUID="1222e77c-6ae5-4ce1-a15f-caa491260730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 30 00:33:51.879948 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.873669 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aabed96e-162a-4e58-a3b1-57750140a0e8-tls-certs" pod "aabed96e-162a-4e58-a3b1-57750140a0e8" (UID: "aabed96e-162a-4e58-a3b1-57750140a0e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:33:51.879948 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.877247 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlvx5\" (UniqueName: \"kubernetes.io/projected/aabed96e-162a-4e58-a3b1-57750140a0e8-kube-api-access-vlvx5\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:33:51.879948 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.877278 5001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aabed96e-162a-4e58-a3b1-57750140a0e8-tls-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:33:51.965153 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.965099 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf200711-0050-4fae-8fe0-385b6e39dbe2-kube-api-access-tjcbr" pod "cf200711-0050-4fae-8fe0-385b6e39dbe2" (UID: "cf200711-0050-4fae-8fe0-385b6e39dbe2"). InnerVolumeSpecName "kube-api-access-tjcbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:33:51.965437 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.965404 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf200711-0050-4fae-8fe0-385b6e39dbe2-maas-api-xks-ca" pod "cf200711-0050-4fae-8fe0-385b6e39dbe2" (UID: "cf200711-0050-4fae-8fe0-385b6e39dbe2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:33:51.988287 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.987478 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjcbr\" (UniqueName: \"kubernetes.io/projected/cf200711-0050-4fae-8fe0-385b6e39dbe2-kube-api-access-tjcbr\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:33:51.988287 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.987517 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf200711-0050-4fae-8fe0-385b6e39dbe2-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:33:52.010007 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:51.999171 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aabed96e-162a-4e58-a3b1-57750140a0e8-dshm" pod "aabed96e-162a-4e58-a3b1-57750140a0e8" (UID: "aabed96e-162a-4e58-a3b1-57750140a0e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:33:52.090547 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:52.089846 5001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aabed96e-162a-4e58-a3b1-57750140a0e8-dshm\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:33:52.638058 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:52.638015 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh"] Oct 30 00:33:52.644732 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:52.644688 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:33:52.644879 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:52.644780 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:33:52.644879 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:52.644845 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:33:52.645376 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:52.645355 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-564c8f-wgm5x" Oct 30 00:33:52.645495 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:52.645468 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-564c8f-wgm5x" event={"ID":"cf200711-0050-4fae-8fe0-385b6e39dbe2","Type":"ContainerDied","Data":"2fa7b2788d53b88158f6f41c86c0dd1a3c2d96dc9975765fe2d141ea4d563741"} Oct 30 00:33:52.645569 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:52.645516 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fa7b2788d53b88158f6f41c86c0dd1a3c2d96dc9975765fe2d141ea4d563741" Oct 30 00:33:52.660817 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:52.658729 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" Oct 30 00:33:52.660817 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:52.658844 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85" event={"ID":"aabed96e-162a-4e58-a3b1-57750140a0e8","Type":"ContainerDied","Data":"2a76875f98957de06abb712ad0e7e21e1020f8b6acaebd4426cf7127b6a40e79"} Oct 30 00:33:52.660817 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:52.658896 5001 scope.go:122] "RemoveContainer" containerID="03bebce622ce78c29cf07a2e18e5c4ff70786e27d839711ea040730c3ca8fd8b" Oct 30 00:33:52.793785 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:52.793697 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85"] Oct 30 00:33:52.802532 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:52.802490 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-b1c718/distinct-2-w5-b1c718-kserve-6b7db4cd77-9xl85"] Oct 30 00:33:52.849098 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:52.839616 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 30 00:33:52.849098 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:52.839686 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" podUID="1222e77c-6ae5-4ce1-a15f-caa491260730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 30 00:33:52.950580 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:52.950487 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" path="/var/lib/kubelet/pods/aabed96e-162a-4e58-a3b1-57750140a0e8/volumes" Oct 30 00:33:53.685472 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:53.684147 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" event={"ID":"ff8dd458-3a31-4084-b71e-2ecb4114c4e0","Type":"ContainerStarted","Data":"90b44cb7797ab9211767e050a7f01e06de111cf75c35871faa72b7c2e1f8587e"} Oct 30 00:33:53.685472 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:53.684218 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" event={"ID":"ff8dd458-3a31-4084-b71e-2ecb4114c4e0","Type":"ContainerStarted","Data":"13d4a1291df03ed72e65fc26db22c36d56283ed8b4750fa62f2a50beaae39b8a"} Oct 30 00:33:53.731023 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:53.729530 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" podStartSLOduration=2.729507413 podStartE2EDuration="2.729507413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:53.728009051 +0000 UTC m=+1748.933510234" watchObservedRunningTime="2026-10-30 00:33:53.729507413 +0000 UTC m=+1748.935008596" Oct 30 00:33:53.944940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:53.943068 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:33:53.946331 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:53.946247 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:33:54.492217 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:54.492177 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:33:54.499669 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:54.499643 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:33:54.701854 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:54.701798 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:33:54.703113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:54.703088 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:33:55.713564 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:55.713470 5001 generic.go:356] "Generic (PLEG): container finished" podID="b41b47fa-39f2-4620-b648-797837a6cd1a" containerID="c18c9fa6e49b4deaa00bf10f962247c1626d4b0e715d446e8281aaabd6755cfe" exitCode=0 Oct 30 00:33:55.713564 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:55.713550 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" event={"ID":"b41b47fa-39f2-4620-b648-797837a6cd1a","Type":"ContainerDied","Data":"c18c9fa6e49b4deaa00bf10f962247c1626d4b0e715d446e8281aaabd6755cfe"} Oct 30 00:33:55.714124 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:55.713957 5001 scope.go:122] "RemoveContainer" containerID="c18c9fa6e49b4deaa00bf10f962247c1626d4b0e715d446e8281aaabd6755cfe" Oct 30 00:33:55.715479 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:55.715460 5001 generic.go:356] "Generic (PLEG): container finished" podID="ff9c1b8a-7343-4566-9611-7dc87cb6fea5" containerID="43b8ad597cf0aaaad4f4de137cc30ef303022bd09c4c21700397e0ae15d15a30" exitCode=0 Oct 30 00:33:55.715587 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:55.715467 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:33:55.715587 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:55.715532 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:33:55.715587 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:55.715542 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" event={"ID":"ff9c1b8a-7343-4566-9611-7dc87cb6fea5","Type":"ContainerDied","Data":"43b8ad597cf0aaaad4f4de137cc30ef303022bd09c4c21700397e0ae15d15a30"} Oct 30 00:33:55.715587 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:55.715565 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:33:55.715974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:55.715949 5001 scope.go:122] "RemoveContainer" containerID="43b8ad597cf0aaaad4f4de137cc30ef303022bd09c4c21700397e0ae15d15a30" Oct 30 00:33:55.717319 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:55.717291 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:33:55.717386 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:55.717353 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:33:55.717428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:55.717390 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:33:56.660079 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.660044 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-778b5858-jq9cm"] Oct 30 00:33:56.671021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.661257 5001 cpu_manager.go:414] "containerMap: removing container" podUID="cf200711-0050-4fae-8fe0-385b6e39dbe2" containerName="revoke-keys" Oct 30 00:33:56.671021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.661286 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf200711-0050-4fae-8fe0-385b6e39dbe2" containerName="revoke-keys" Oct 30 00:33:56.671021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.661330 5001 cpu_manager.go:414] "containerMap: removing container" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" Oct 30 00:33:56.671021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.661337 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" Oct 30 00:33:56.671021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.661532 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="aabed96e-162a-4e58-a3b1-57750140a0e8" containerName="main" Oct 30 00:33:56.671021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.661556 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf200711-0050-4fae-8fe0-385b6e39dbe2" containerName="revoke-keys" Oct 30 00:33:56.686957 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.683034 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-778b5858-jq9cm" Oct 30 00:33:56.691423 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.688199 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cee150a0-qkd8t"] Oct 30 00:33:56.692054 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.692036 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee150a0-qkd8t" Oct 30 00:33:56.696239 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.695976 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:33:56.696239 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.696069 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:33:56.696239 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.696135 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:33:56.696239 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.696211 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:33:56.699889 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.699870 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-778b5858-jq9cm"] Oct 30 00:33:56.732933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.707895 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cee150a0-qkd8t"] Oct 30 00:33:56.783242 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.783204 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" event={"ID":"b41b47fa-39f2-4620-b648-797837a6cd1a","Type":"ContainerStarted","Data":"cbf662eb37a2c8bb40b8c72ae27a2fd53844eecfe68b4d677430646c3591f02b"} Oct 30 00:33:56.786185 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.785627 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" event={"ID":"ff9c1b8a-7343-4566-9611-7dc87cb6fea5","Type":"ContainerStarted","Data":"548eb7c34ab72826bbc91fc0d93c7a7ef17404127406a98538ae295715ac3f5f"} Oct 30 00:33:56.866941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.862352 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n67p8\" (UniqueName: \"kubernetes.io/projected/2a4eee95-140b-45a8-a9a8-0c5b2be11e65-kube-api-access-n67p8\") pod \"maas-api-revoke-sub-cee150a0-qkd8t\" (UID: \"2a4eee95-140b-45a8-a9a8-0c5b2be11e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee150a0-qkd8t" Oct 30 00:33:56.866941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.862446 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b492299d-9c08-491b-bcd8-3d4ae73f5522-maas-api-xks-ca\") pod \"maas-api-revoke-sub-778b5858-jq9cm\" (UID: \"b492299d-9c08-491b-bcd8-3d4ae73f5522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-778b5858-jq9cm" Oct 30 00:33:56.866941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.862490 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a4eee95-140b-45a8-a9a8-0c5b2be11e65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cee150a0-qkd8t\" (UID: \"2a4eee95-140b-45a8-a9a8-0c5b2be11e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee150a0-qkd8t" Oct 30 00:33:56.866941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.862603 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b492299d-9c08-491b-bcd8-3d4ae73f5522-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-778b5858-jq9cm\" (UID: \"b492299d-9c08-491b-bcd8-3d4ae73f5522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-778b5858-jq9cm" Oct 30 00:33:56.866941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.862664 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a4eee95-140b-45a8-a9a8-0c5b2be11e65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cee150a0-qkd8t\" (UID: \"2a4eee95-140b-45a8-a9a8-0c5b2be11e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee150a0-qkd8t" Oct 30 00:33:56.866941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.862818 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnn2q\" (UniqueName: \"kubernetes.io/projected/b492299d-9c08-491b-bcd8-3d4ae73f5522-kube-api-access-xnn2q\") pod \"maas-api-revoke-sub-778b5858-jq9cm\" (UID: \"b492299d-9c08-491b-bcd8-3d4ae73f5522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-778b5858-jq9cm" Oct 30 00:33:56.964818 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.964724 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b492299d-9c08-491b-bcd8-3d4ae73f5522-maas-api-xks-ca\") pod \"maas-api-revoke-sub-778b5858-jq9cm\" (UID: \"b492299d-9c08-491b-bcd8-3d4ae73f5522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-778b5858-jq9cm" Oct 30 00:33:56.964818 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.964808 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a4eee95-140b-45a8-a9a8-0c5b2be11e65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cee150a0-qkd8t\" (UID: \"2a4eee95-140b-45a8-a9a8-0c5b2be11e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee150a0-qkd8t" Oct 30 00:33:56.965109 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.965086 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b492299d-9c08-491b-bcd8-3d4ae73f5522-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-778b5858-jq9cm\" (UID: \"b492299d-9c08-491b-bcd8-3d4ae73f5522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-778b5858-jq9cm" Oct 30 00:33:56.965199 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.965177 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a4eee95-140b-45a8-a9a8-0c5b2be11e65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cee150a0-qkd8t\" (UID: \"2a4eee95-140b-45a8-a9a8-0c5b2be11e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee150a0-qkd8t" Oct 30 00:33:56.965524 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.965491 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnn2q\" (UniqueName: \"kubernetes.io/projected/b492299d-9c08-491b-bcd8-3d4ae73f5522-kube-api-access-xnn2q\") pod \"maas-api-revoke-sub-778b5858-jq9cm\" (UID: \"b492299d-9c08-491b-bcd8-3d4ae73f5522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-778b5858-jq9cm" Oct 30 00:33:56.966289 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.965729 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n67p8\" (UniqueName: \"kubernetes.io/projected/2a4eee95-140b-45a8-a9a8-0c5b2be11e65-kube-api-access-n67p8\") pod \"maas-api-revoke-sub-cee150a0-qkd8t\" (UID: \"2a4eee95-140b-45a8-a9a8-0c5b2be11e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee150a0-qkd8t" Oct 30 00:33:56.966289 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.965956 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b492299d-9c08-491b-bcd8-3d4ae73f5522-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-778b5858-jq9cm\" (UID: \"b492299d-9c08-491b-bcd8-3d4ae73f5522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-778b5858-jq9cm" Oct 30 00:33:56.966448 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.966289 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a4eee95-140b-45a8-a9a8-0c5b2be11e65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cee150a0-qkd8t\" (UID: \"2a4eee95-140b-45a8-a9a8-0c5b2be11e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee150a0-qkd8t" Oct 30 00:33:56.967957 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.967933 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b492299d-9c08-491b-bcd8-3d4ae73f5522-maas-api-xks-ca\") pod \"maas-api-revoke-sub-778b5858-jq9cm\" (UID: \"b492299d-9c08-491b-bcd8-3d4ae73f5522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-778b5858-jq9cm" Oct 30 00:33:56.968426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.968408 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a4eee95-140b-45a8-a9a8-0c5b2be11e65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cee150a0-qkd8t\" (UID: \"2a4eee95-140b-45a8-a9a8-0c5b2be11e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee150a0-qkd8t" Oct 30 00:33:56.988830 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.988805 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnn2q\" (UniqueName: \"kubernetes.io/projected/b492299d-9c08-491b-bcd8-3d4ae73f5522-kube-api-access-xnn2q\") pod \"maas-api-revoke-sub-778b5858-jq9cm\" (UID: \"b492299d-9c08-491b-bcd8-3d4ae73f5522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-778b5858-jq9cm" Oct 30 00:33:56.993303 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:56.993278 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n67p8\" (UniqueName: \"kubernetes.io/projected/2a4eee95-140b-45a8-a9a8-0c5b2be11e65-kube-api-access-n67p8\") pod \"maas-api-revoke-sub-cee150a0-qkd8t\" (UID: \"2a4eee95-140b-45a8-a9a8-0c5b2be11e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee150a0-qkd8t" Oct 30 00:33:57.005179 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:57.005154 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-778b5858-jq9cm" Oct 30 00:33:57.075504 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:57.075472 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee150a0-qkd8t" Oct 30 00:33:57.462468 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:57.462434 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:33:57.462832 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:57.462809 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 30 00:33:57.463304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:57.463008 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" podUID="ff9c1b8a-7343-4566-9611-7dc87cb6fea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 30 00:33:57.687012 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:57.686973 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-564c8f-57dbdd8c55-q6gz2"] Oct 30 00:33:57.791643 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:57.786418 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:33:57.791643 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:57.786565 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 30 00:33:57.791643 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:57.786612 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" podUID="b41b47fa-39f2-4620-b648-797837a6cd1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 30 00:33:58.162948 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:58.152097 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cee150a0-qkd8t"] Oct 30 00:33:58.173963 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:58.166211 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee150a0-qkd8t" event={"ID":"2a4eee95-140b-45a8-a9a8-0c5b2be11e65","Type":"ContainerStarted","Data":"efc2fc05cba68f501628ab9be543f5cdb752e9a31b193503845e4a9d2a76b947"} Oct 30 00:33:58.369128 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:33:58.369092 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb492299d_9c08_491b_bcd8_3d4ae73f5522.slice/crio-5d2e5ce11aeeb26fb137807d8e5043d9a51a129133ad0a156f7ad3e8e04b3540 WatchSource:0}: Error finding container 5d2e5ce11aeeb26fb137807d8e5043d9a51a129133ad0a156f7ad3e8e04b3540: Status 404 returned error can't find the container with id 5d2e5ce11aeeb26fb137807d8e5043d9a51a129133ad0a156f7ad3e8e04b3540 Oct 30 00:33:58.504068 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:58.503956 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-778b5858-jq9cm"] Oct 30 00:33:58.553002 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:58.552968 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ecb34a-7ffdbf96c-8lgp2" Oct 30 00:33:58.553792 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:58.513193 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 30 00:33:58.554039 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:58.554011 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" podUID="ff9c1b8a-7343-4566-9611-7dc87cb6fea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 30 00:33:58.829946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:58.824996 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 30 00:33:58.829946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:58.825083 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" podUID="b41b47fa-39f2-4620-b648-797837a6cd1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 30 00:33:59.400599 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:59.400570 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-415d24-8cc6f76cf-vnt5d" Oct 30 00:33:59.401039 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:59.401015 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ecb34a-7c4f79cf86-6l8nz" Oct 30 00:33:59.401280 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:59.401264 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-05c6b2-55b55d747-dgrp9" Oct 30 00:33:59.406260 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:59.406239 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-778b5858-jq9cm" event={"ID":"b492299d-9c08-491b-bcd8-3d4ae73f5522","Type":"ContainerStarted","Data":"5d2e5ce11aeeb26fb137807d8e5043d9a51a129133ad0a156f7ad3e8e04b3540"} Oct 30 00:33:59.477851 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:59.477036 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 30 00:33:59.477851 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:59.477100 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" podUID="ff9c1b8a-7343-4566-9611-7dc87cb6fea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 30 00:33:59.597076 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:59.597047 5001 scope.go:122] "RemoveContainer" containerID="f10fe7ca539096970b05937017c10fc1e492f75a6174b548cc05fd9cfd2c712b" Oct 30 00:33:59.797933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:59.790343 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 30 00:33:59.797933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:33:59.790402 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" podUID="b41b47fa-39f2-4620-b648-797837a6cd1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 30 00:34:00.373775 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:00.352543 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-415d24-9fd8c87c8-wn6vg" Oct 30 00:34:00.422097 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:00.422009 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f1268c-7745d464f5-hf7k2"] Oct 30 00:34:00.480813 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:00.476898 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 30 00:34:00.480813 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:00.476970 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" podUID="ff9c1b8a-7343-4566-9611-7dc87cb6fea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 30 00:34:00.841282 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:00.800879 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f1268c-7745d464f5-hf7k2"] Oct 30 00:34:00.841282 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:00.801047 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f1268c-7745d464f5-hf7k2" Oct 30 00:34:00.844083 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:00.844039 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 30 00:34:00.866397 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:00.866351 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" podUID="b41b47fa-39f2-4620-b648-797837a6cd1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 30 00:34:00.932027 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:00.890057 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:01.038388 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:01.033171 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:01.069867 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:01.069818 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxvxd\" (UniqueName: \"kubernetes.io/projected/63af7822-1fdf-4af9-977b-11ce91f5ec63-kube-api-access-qxvxd\") pod \"payload-pre-processing-e2e-shared-b-w6-f1268c-7745d464f5-hf7k2\" (UID: \"63af7822-1fdf-4af9-977b-11ce91f5ec63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f1268c-7745d464f5-hf7k2" Oct 30 00:34:01.070387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:01.070351 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63af7822-1fdf-4af9-977b-11ce91f5ec63-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-f1268c-7745d464f5-hf7k2\" (UID: \"63af7822-1fdf-4af9-977b-11ce91f5ec63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f1268c-7745d464f5-hf7k2" Oct 30 00:34:01.093013 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:01.092995 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-f1268c-dockercfg-j69rr\"" Oct 30 00:34:01.100905 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:01.100354 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee150a0-qkd8t" event={"ID":"2a4eee95-140b-45a8-a9a8-0c5b2be11e65","Type":"ContainerStarted","Data":"92be57859ada05e4f289424fb9d02a42a08f5710838ec08a7a374739664c9d3a"} Oct 30 00:34:01.101596 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:01.101576 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-778b5858-jq9cm" event={"ID":"b492299d-9c08-491b-bcd8-3d4ae73f5522","Type":"ContainerStarted","Data":"8e1217c5d6d8f8035905748e4834e9f91f224196f01c80ffa524a945a026ad68"} Oct 30 00:34:01.103674 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:01.103642 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-f1268c\"" Oct 30 00:34:01.172737 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:01.172701 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxvxd\" (UniqueName: \"kubernetes.io/projected/63af7822-1fdf-4af9-977b-11ce91f5ec63-kube-api-access-qxvxd\") pod \"payload-pre-processing-e2e-shared-b-w6-f1268c-7745d464f5-hf7k2\" (UID: \"63af7822-1fdf-4af9-977b-11ce91f5ec63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f1268c-7745d464f5-hf7k2" Oct 30 00:34:01.172957 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:01.172883 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63af7822-1fdf-4af9-977b-11ce91f5ec63-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-f1268c-7745d464f5-hf7k2\" (UID: \"63af7822-1fdf-4af9-977b-11ce91f5ec63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f1268c-7745d464f5-hf7k2" Oct 30 00:34:01.173660 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:01.173636 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63af7822-1fdf-4af9-977b-11ce91f5ec63-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-f1268c-7745d464f5-hf7k2\" (UID: \"63af7822-1fdf-4af9-977b-11ce91f5ec63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f1268c-7745d464f5-hf7k2" Oct 30 00:34:01.193261 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:01.193237 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxvxd\" (UniqueName: \"kubernetes.io/projected/63af7822-1fdf-4af9-977b-11ce91f5ec63-kube-api-access-qxvxd\") pod \"payload-pre-processing-e2e-shared-b-w6-f1268c-7745d464f5-hf7k2\" (UID: \"63af7822-1fdf-4af9-977b-11ce91f5ec63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f1268c-7745d464f5-hf7k2" Oct 30 00:34:01.266903 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:01.266870 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-05c6b2-6f8558fd8b-kslr7" Oct 30 00:34:01.462839 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:01.462745 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 30 00:34:01.463342 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:01.462815 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" podUID="ff9c1b8a-7343-4566-9611-7dc87cb6fea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 30 00:34:01.489286 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:01.489248 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f1268c-7745d464f5-hf7k2" Oct 30 00:34:01.777516 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:01.777488 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 30 00:34:01.777653 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:01.777556 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" podUID="b41b47fa-39f2-4620-b648-797837a6cd1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 30 00:34:01.974952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:01.974882 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cef097b1-f2wkl"] Oct 30 00:34:01.981555 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:01.980225 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef097b1-f2wkl" Oct 30 00:34:01.986969 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:01.986946 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f18e88b3-26p8b"] Oct 30 00:34:01.992932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:01.992884 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18e88b3-26p8b" Oct 30 00:34:01.997312 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:01.997295 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cef097b1-f2wkl"] Oct 30 00:34:02.095494 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:02.093951 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f18e88b3-26p8b"] Oct 30 00:34:02.228991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:02.219529 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tt7l\" (UniqueName: \"kubernetes.io/projected/3665be1a-5bbd-49aa-b414-8d168419bc71-kube-api-access-4tt7l\") pod \"maas-api-revoke-sub-cef097b1-f2wkl\" (UID: \"3665be1a-5bbd-49aa-b414-8d168419bc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef097b1-f2wkl" Oct 30 00:34:02.228991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:02.219894 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3665be1a-5bbd-49aa-b414-8d168419bc71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cef097b1-f2wkl\" (UID: \"3665be1a-5bbd-49aa-b414-8d168419bc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef097b1-f2wkl" Oct 30 00:34:02.228991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:02.219959 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f18e88b3-26p8b\" (UID: \"f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18e88b3-26p8b" Oct 30 00:34:02.228991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:02.219985 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3665be1a-5bbd-49aa-b414-8d168419bc71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cef097b1-f2wkl\" (UID: \"3665be1a-5bbd-49aa-b414-8d168419bc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef097b1-f2wkl" Oct 30 00:34:02.228991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:02.220048 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pr28\" (UniqueName: \"kubernetes.io/projected/f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d-kube-api-access-8pr28\") pod \"maas-api-revoke-sub-f18e88b3-26p8b\" (UID: \"f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18e88b3-26p8b" Oct 30 00:34:02.228991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:02.220082 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f18e88b3-26p8b\" (UID: \"f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18e88b3-26p8b" Oct 30 00:34:02.366080 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:02.331026 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f1268c-7745d464f5-hf7k2"] Oct 30 00:34:02.443721 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:02.443681 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3665be1a-5bbd-49aa-b414-8d168419bc71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cef097b1-f2wkl\" (UID: \"3665be1a-5bbd-49aa-b414-8d168419bc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef097b1-f2wkl" Oct 30 00:34:02.484486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:02.484444 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f18e88b3-26p8b\" (UID: \"f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18e88b3-26p8b" Oct 30 00:34:02.485102 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:02.485054 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3665be1a-5bbd-49aa-b414-8d168419bc71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cef097b1-f2wkl\" (UID: \"3665be1a-5bbd-49aa-b414-8d168419bc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef097b1-f2wkl" Oct 30 00:34:02.485363 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:02.485343 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pr28\" (UniqueName: \"kubernetes.io/projected/f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d-kube-api-access-8pr28\") pod \"maas-api-revoke-sub-f18e88b3-26p8b\" (UID: \"f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18e88b3-26p8b" Oct 30 00:34:02.485548 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:02.485520 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f18e88b3-26p8b\" (UID: \"f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18e88b3-26p8b" Oct 30 00:34:02.485843 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:02.485822 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tt7l\" (UniqueName: \"kubernetes.io/projected/3665be1a-5bbd-49aa-b414-8d168419bc71-kube-api-access-4tt7l\") pod \"maas-api-revoke-sub-cef097b1-f2wkl\" (UID: \"3665be1a-5bbd-49aa-b414-8d168419bc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef097b1-f2wkl" Oct 30 00:34:02.486686 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:02.475417 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 30 00:34:02.501623 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:02.501586 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" podUID="ff9c1b8a-7343-4566-9611-7dc87cb6fea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 30 00:34:02.501828 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:02.487796 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f18e88b3-26p8b\" (UID: \"f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18e88b3-26p8b" Oct 30 00:34:02.502074 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:02.458826 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3665be1a-5bbd-49aa-b414-8d168419bc71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cef097b1-f2wkl\" (UID: \"3665be1a-5bbd-49aa-b414-8d168419bc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef097b1-f2wkl" Oct 30 00:34:02.545682 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:02.545637 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tt7l\" (UniqueName: \"kubernetes.io/projected/3665be1a-5bbd-49aa-b414-8d168419bc71-kube-api-access-4tt7l\") pod \"maas-api-revoke-sub-cef097b1-f2wkl\" (UID: \"3665be1a-5bbd-49aa-b414-8d168419bc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef097b1-f2wkl" Oct 30 00:34:02.634201 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:02.634114 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f18e88b3-26p8b\" (UID: \"f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18e88b3-26p8b" Oct 30 00:34:02.643300 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:02.643265 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pr28\" (UniqueName: \"kubernetes.io/projected/f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d-kube-api-access-8pr28\") pod \"maas-api-revoke-sub-f18e88b3-26p8b\" (UID: \"f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18e88b3-26p8b" Oct 30 00:34:02.706078 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:02.692172 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-564c8f-57dbdd8c55-q6gz2" podUID="2077ddd9-c805-453e-8611-fc8b90b3b3c1" containerName="payload-pre-processing" containerID="cri-o://96a3d3e90bb2f7db6d7bd8518d9bd7080b09a4f9f963d7681f36980417f8d102" gracePeriod=25 Oct 30 00:34:02.706078 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:02.702691 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3665be1a-5bbd-49aa-b414-8d168419bc71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cef097b1-f2wkl\" (UID: \"3665be1a-5bbd-49aa-b414-8d168419bc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef097b1-f2wkl" Oct 30 00:34:02.707884 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:02.707673 5001 generic.go:356] "Generic (PLEG): container finished" podID="2a4eee95-140b-45a8-a9a8-0c5b2be11e65" containerID="92be57859ada05e4f289424fb9d02a42a08f5710838ec08a7a374739664c9d3a" exitCode=0 Oct 30 00:34:02.727006 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:02.711228 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef097b1-f2wkl" Oct 30 00:34:02.728038 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:02.727646 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee150a0-qkd8t" event={"ID":"2a4eee95-140b-45a8-a9a8-0c5b2be11e65","Type":"ContainerDied","Data":"92be57859ada05e4f289424fb9d02a42a08f5710838ec08a7a374739664c9d3a"} Oct 30 00:34:02.728038 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:02.727979 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18e88b3-26p8b" Oct 30 00:34:02.802954 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:02.788856 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 30 00:34:02.802954 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:02.788934 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" podUID="b41b47fa-39f2-4620-b648-797837a6cd1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 30 00:34:03.005533 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:03.005494 5001 generic.go:356] "Generic (PLEG): container finished" podID="b492299d-9c08-491b-bcd8-3d4ae73f5522" containerID="8e1217c5d6d8f8035905748e4834e9f91f224196f01c80ffa524a945a026ad68" exitCode=0 Oct 30 00:34:03.232506 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:03.232091 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-564c8f-57dbdd8c55-q6gz2" podUID="2077ddd9-c805-453e-8611-fc8b90b3b3c1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:34:03.436725 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:03.436680 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-778b5858-jq9cm" event={"ID":"b492299d-9c08-491b-bcd8-3d4ae73f5522","Type":"ContainerDied","Data":"8e1217c5d6d8f8035905748e4834e9f91f224196f01c80ffa524a945a026ad68"} Oct 30 00:34:03.437033 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:03.437008 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f1268c-7745d464f5-hf7k2" event={"ID":"63af7822-1fdf-4af9-977b-11ce91f5ec63","Type":"ContainerStarted","Data":"4fe68158eaa7ab62b1392c713addc8ed533e5df47220b66c551ef51ca3c22890"} Oct 30 00:34:03.540682 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:03.540647 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 30 00:34:03.558052 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:03.558001 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" podUID="ff9c1b8a-7343-4566-9611-7dc87cb6fea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 30 00:34:03.760506 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:03.760477 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:34:03.778306 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:03.778276 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 30 00:34:03.778577 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:03.778544 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" podUID="b41b47fa-39f2-4620-b648-797837a6cd1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 30 00:34:04.068031 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:04.068001 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee150a0-qkd8t" Oct 30 00:34:04.180944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:04.179080 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:04.180944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:04.179446 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:34:04.211090 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:04.211040 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:04.212231 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:04.211773 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a4eee95-140b-45a8-a9a8-0c5b2be11e65-kube-api-access-n67p8\" (UniqueName: \"kubernetes.io/projected/2a4eee95-140b-45a8-a9a8-0c5b2be11e65-kube-api-access-n67p8\") pod \"2a4eee95-140b-45a8-a9a8-0c5b2be11e65\" (UID: \"2a4eee95-140b-45a8-a9a8-0c5b2be11e65\") " Oct 30 00:34:04.226009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:04.225341 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a4eee95-140b-45a8-a9a8-0c5b2be11e65-maas-api-openshift-ca" pod "2a4eee95-140b-45a8-a9a8-0c5b2be11e65" (UID: "2a4eee95-140b-45a8-a9a8-0c5b2be11e65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:34:04.312952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:04.311686 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a4eee95-140b-45a8-a9a8-0c5b2be11e65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a4eee95-140b-45a8-a9a8-0c5b2be11e65-maas-api-openshift-ca\") pod \"2a4eee95-140b-45a8-a9a8-0c5b2be11e65\" (UID: \"2a4eee95-140b-45a8-a9a8-0c5b2be11e65\") " Oct 30 00:34:04.312952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:04.312325 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a4eee95-140b-45a8-a9a8-0c5b2be11e65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a4eee95-140b-45a8-a9a8-0c5b2be11e65-maas-api-xks-ca\") pod \"2a4eee95-140b-45a8-a9a8-0c5b2be11e65\" (UID: \"2a4eee95-140b-45a8-a9a8-0c5b2be11e65\") " Oct 30 00:34:04.361971 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:04.328097 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a4eee95-140b-45a8-a9a8-0c5b2be11e65-kube-api-access-n67p8" pod "2a4eee95-140b-45a8-a9a8-0c5b2be11e65" (UID: "2a4eee95-140b-45a8-a9a8-0c5b2be11e65"). InnerVolumeSpecName "kube-api-access-n67p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:04.361971 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:04.361604 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a4eee95-140b-45a8-a9a8-0c5b2be11e65-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:04.361971 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:04.361640 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n67p8\" (UniqueName: \"kubernetes.io/projected/2a4eee95-140b-45a8-a9a8-0c5b2be11e65-kube-api-access-n67p8\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:04.619327 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:04.547151 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a4eee95-140b-45a8-a9a8-0c5b2be11e65-maas-api-xks-ca" pod "2a4eee95-140b-45a8-a9a8-0c5b2be11e65" (UID: "2a4eee95-140b-45a8-a9a8-0c5b2be11e65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:34:04.619327 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:04.557822 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a4eee95-140b-45a8-a9a8-0c5b2be11e65-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:04.619327 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:04.568465 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 30 00:34:04.619327 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:04.568523 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" podUID="ff9c1b8a-7343-4566-9611-7dc87cb6fea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 30 00:34:04.621687 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:04.621643 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 30 00:34:04.664721 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:04.664679 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 00:34:04.864958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:04.784490 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 30 00:34:04.865305 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:04.865244 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" podUID="b41b47fa-39f2-4620-b648-797837a6cd1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 30 00:34:04.865853 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:04.807506 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f1268c-7745d464f5-hf7k2" event={"ID":"63af7822-1fdf-4af9-977b-11ce91f5ec63","Type":"ContainerStarted","Data":"390dcbf435e8ed1f319786b287313efb5259dae2a4cd01804706982bdc805bf1"} Oct 30 00:34:04.866060 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:04.866043 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f1268c-7745d464f5-hf7k2" Oct 30 00:34:05.000071 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:04.909468 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:05.000071 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:04.954064 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f1268c-7745d464f5-hf7k2" podStartSLOduration=4.954039859 podStartE2EDuration="4.954039859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:04.896671968 +0000 UTC m=+1760.102173146" watchObservedRunningTime="2026-10-30 00:34:04.954039859 +0000 UTC m=+1760.159541043" Oct 30 00:34:05.093086 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:05.029041 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:05.297699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:05.297656 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee150a0-qkd8t" Oct 30 00:34:05.402958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:05.325306 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:05.402958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:05.330275 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:05.402958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:05.357613 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:34:05.402958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:05.365737 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee150a0-qkd8t" event={"ID":"2a4eee95-140b-45a8-a9a8-0c5b2be11e65","Type":"ContainerDied","Data":"efc2fc05cba68f501628ab9be543f5cdb752e9a31b193503845e4a9d2a76b947"} Oct 30 00:34:05.402958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:05.365812 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efc2fc05cba68f501628ab9be543f5cdb752e9a31b193503845e4a9d2a76b947" Oct 30 00:34:05.402958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:05.365875 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:34:05.402958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:05.365933 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:34:05.522945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:05.521315 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 30 00:34:05.522945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:05.521381 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" podUID="ff9c1b8a-7343-4566-9611-7dc87cb6fea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 30 00:34:05.654994 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:05.645779 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:05.654994 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:05.651105 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:34:05.668321 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:05.668273 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:05.670167 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:05.670138 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:34:05.823600 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:05.823549 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 30 00:34:05.823900 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:05.823867 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" podUID="b41b47fa-39f2-4620-b648-797837a6cd1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 30 00:34:05.826576 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:05.826550 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:34:05.935993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:05.934158 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 00:34:05.935993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:05.934751 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:34:06.166086 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:06.166028 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:06.167070 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:06.167046 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:34:06.352775 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:06.332690 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" containerID="cri-o://693cf671fd5479bf07b7de17356b2f2d2e8e643f67e2b1d405ef4677a1af11ce" gracePeriod=45 Oct 30 00:34:06.352775 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:06.344870 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:06.352775 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:06.348803 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:06.364128 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:06.363036 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:06.364128 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:06.363563 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:34:06.464536 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:06.464478 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 30 00:34:06.464860 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:06.464810 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" podUID="ff9c1b8a-7343-4566-9611-7dc87cb6fea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 30 00:34:06.686075 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:06.686011 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:06.769167 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:06.686676 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:06.789960 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:06.776008 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 30 00:34:06.789960 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:06.776063 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" podUID="b41b47fa-39f2-4620-b648-797837a6cd1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 30 00:34:06.978230 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:06.866023 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:34:06.992621 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:06.990013 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 00:34:07.085350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:07.085267 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": context deadline exceeded" Oct 30 00:34:07.216922 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:07.216844 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:07.376165 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:07.358159 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:07.413955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:07.388069 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:34:07.524013 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:07.512534 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 30 00:34:07.537158 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:07.537100 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" podUID="ff9c1b8a-7343-4566-9611-7dc87cb6fea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 30 00:34:07.767213 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:07.765166 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:07.770572 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:07.770360 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:07.805106 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:07.804600 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 30 00:34:07.805106 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:07.804670 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" podUID="b41b47fa-39f2-4620-b648-797837a6cd1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 30 00:34:08.016107 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:08.014065 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:08.103641 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:34:08.068137 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3665be1a_5bbd_49aa_b414_8d168419bc71.slice/crio-06a74176a0af455f4a7ae0724ddae2a25d4320b6e71695aa38f00e4137d81d4e WatchSource:0}: Error finding container 06a74176a0af455f4a7ae0724ddae2a25d4320b6e71695aa38f00e4137d81d4e: Status 404 returned error can't find the container with id 06a74176a0af455f4a7ae0724ddae2a25d4320b6e71695aa38f00e4137d81d4e Oct 30 00:34:08.135242 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:34:08.122775 5001 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.2s" Oct 30 00:34:08.135242 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:08.134944 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-778b5858-jq9cm" Oct 30 00:34:08.226318 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:08.226189 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b492299d-9c08-491b-bcd8-3d4ae73f5522-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b492299d-9c08-491b-bcd8-3d4ae73f5522-maas-api-openshift-ca\") pod \"b492299d-9c08-491b-bcd8-3d4ae73f5522\" (UID: \"b492299d-9c08-491b-bcd8-3d4ae73f5522\") " Oct 30 00:34:08.252984 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:08.220886 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:34:08.374290 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:08.338017 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b492299d-9c08-491b-bcd8-3d4ae73f5522-kube-api-access-xnn2q\" (UniqueName: \"kubernetes.io/projected/b492299d-9c08-491b-bcd8-3d4ae73f5522-kube-api-access-xnn2q\") pod \"b492299d-9c08-491b-bcd8-3d4ae73f5522\" (UID: \"b492299d-9c08-491b-bcd8-3d4ae73f5522\") " Oct 30 00:34:08.374290 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:08.338079 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b492299d-9c08-491b-bcd8-3d4ae73f5522-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b492299d-9c08-491b-bcd8-3d4ae73f5522-maas-api-xks-ca\") pod \"b492299d-9c08-491b-bcd8-3d4ae73f5522\" (UID: \"b492299d-9c08-491b-bcd8-3d4ae73f5522\") " Oct 30 00:34:08.374290 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:08.357625 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b492299d-9c08-491b-bcd8-3d4ae73f5522-maas-api-openshift-ca" pod "b492299d-9c08-491b-bcd8-3d4ae73f5522" (UID: "b492299d-9c08-491b-bcd8-3d4ae73f5522"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:34:08.374290 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:08.358284 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:34:08.374290 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:08.366782 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b492299d-9c08-491b-bcd8-3d4ae73f5522-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:08.400060 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:08.400009 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:08.424946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:08.401977 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:08.428618 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:34:08.428578 5001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4eee95_140b_45a8_a9a8_0c5b2be11e65.slice\": RecentStats: unable to find data in memory cache]" Oct 30 00:34:08.595389 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:08.595349 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f18e88b3-26p8b"] Oct 30 00:34:08.774098 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:08.774043 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 00:34:08.832997 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:08.829799 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:34:08.832997 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:08.800047 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:08.858956 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:08.858888 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef097b1-f2wkl" event={"ID":"3665be1a-5bbd-49aa-b414-8d168419bc71","Type":"ContainerStarted","Data":"06a74176a0af455f4a7ae0724ddae2a25d4320b6e71695aa38f00e4137d81d4e"} Oct 30 00:34:08.921440 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:08.905133 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:08.923427 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:08.923393 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18e88b3-26p8b" event={"ID":"f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d","Type":"ContainerStarted","Data":"407905dd4cda5d802f4e01fda7b73bcc30b7a9d1a30c273688fa845b7f28f7de"} Oct 30 00:34:09.003814 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:09.003784 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-778b5858-jq9cm" Oct 30 00:34:09.055278 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:09.055237 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 00:34:09.130616 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:09.129593 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 30 00:34:09.130616 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:09.129655 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" podUID="b41b47fa-39f2-4620-b648-797837a6cd1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 30 00:34:09.141960 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:09.137030 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b492299d-9c08-491b-bcd8-3d4ae73f5522-maas-api-xks-ca" pod "b492299d-9c08-491b-bcd8-3d4ae73f5522" (UID: "b492299d-9c08-491b-bcd8-3d4ae73f5522"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:34:09.142676 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:09.142348 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 30 00:34:09.142676 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:09.142402 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" podUID="ff9c1b8a-7343-4566-9611-7dc87cb6fea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 30 00:34:09.291204 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:09.291151 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b492299d-9c08-491b-bcd8-3d4ae73f5522-kube-api-access-xnn2q" pod "b492299d-9c08-491b-bcd8-3d4ae73f5522" (UID: "b492299d-9c08-491b-bcd8-3d4ae73f5522"). InnerVolumeSpecName "kube-api-access-xnn2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:09.319937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:09.292809 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnn2q\" (UniqueName: \"kubernetes.io/projected/b492299d-9c08-491b-bcd8-3d4ae73f5522-kube-api-access-xnn2q\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:09.320233 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:09.320202 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b492299d-9c08-491b-bcd8-3d4ae73f5522-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:09.380478 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:09.380374 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:09.428784 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:09.381013 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": context deadline exceeded" Oct 30 00:34:09.461941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:09.460372 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:09.640783 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:09.636782 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:34:09.934146 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:09.930888 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": read tcp 10.132.0.2:57134->10.132.0.95:8000: read: connection reset by peer" Oct 30 00:34:09.934146 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:09.931092 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:09.934146 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:09.931441 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 00:34:10.067962 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:34:10.053392 5001 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.123s" Oct 30 00:34:10.067962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:10.053437 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cef097b1-f2wkl"] Oct 30 00:34:10.067962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:10.053571 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:34:10.067962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:10.053598 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-778b5858-jq9cm" event={"ID":"b492299d-9c08-491b-bcd8-3d4ae73f5522","Type":"ContainerDied","Data":"5d2e5ce11aeeb26fb137807d8e5043d9a51a129133ad0a156f7ad3e8e04b3540"} Oct 30 00:34:10.067962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:10.053622 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d2e5ce11aeeb26fb137807d8e5043d9a51a129133ad0a156f7ad3e8e04b3540" Oct 30 00:34:10.082935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:10.071371 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:10.312364 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:10.312322 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:34:10.366106 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:10.366068 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": dial tcp 10.132.0.95:8000: connect: connection refused" Oct 30 00:34:10.370105 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:10.370079 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:34:10.432315 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:10.432269 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:34:10.463106 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:10.463062 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:10.574994 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:10.574382 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:34:10.880657 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:10.880551 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:34:10.880937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:10.880890 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" podUID="b41b47fa-39f2-4620-b648-797837a6cd1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:10.911248 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:10.911201 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:10.958556 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:10.955896 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 30 00:34:10.959241 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:10.959203 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 00:34:11.215935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:11.214745 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:11.355967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:11.355904 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": dial tcp 10.132.0.95:8000: connect: connection refused" Oct 30 00:34:11.446053 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:11.445954 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:11.463792 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:11.463750 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:11.585330 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:11.585293 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:34:11.585594 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:11.585565 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" podUID="d34b94b7-c280-4d34-a4b3-5ae295c595cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:11.972150 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:11.898696 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:34:11.972150 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:11.898758 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" podUID="b41b47fa-39f2-4620-b648-797837a6cd1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:11.972150 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:11.922701 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:11.972150 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:11.933357 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:34:11.972150 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:11.959337 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:11.972150 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:11.959477 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:12.080560 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:11.978354 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:34:12.159942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:12.159004 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:12.159942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:12.159877 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18e88b3-26p8b" event={"ID":"f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d","Type":"ContainerStarted","Data":"7db5d89a112e12eba811e2bbf825c08632fd0e712d7f6c943a219811976d0e80"} Oct 30 00:34:12.235433 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:12.235335 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:12.381948 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:12.381740 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": dial tcp 10.132.0.95:8000: connect: connection refused" Oct 30 00:34:12.395357 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:12.393899 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:34:12.459036 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:12.451204 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:12.472794 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:12.472047 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 00:34:12.502067 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:12.501161 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:34:12.734641 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:12.734580 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18e88b3-26p8b" podStartSLOduration=11.734558388 podStartE2EDuration="11.734558388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:12.270358923 +0000 UTC m=+1767.475860129" watchObservedRunningTime="2026-10-30 00:34:12.734558388 +0000 UTC m=+1767.940059590" Oct 30 00:34:12.844381 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:12.842509 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:34:13.004390 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:13.004335 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 00:34:13.015976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:13.007797 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 30 00:34:13.530791 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:13.468394 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Oct 30 00:34:13.625395 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:13.624933 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-564c8f-57dbdd8c55-q6gz2" podUID="2077ddd9-c805-453e-8611-fc8b90b3b3c1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:34:14.000224 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.000134 5001 generic.go:356] "Generic (PLEG): container finished" podID="3665be1a-5bbd-49aa-b414-8d168419bc71" containerID="76a93fc95f464804448af3c16a6ffcec1dedc2d80aaaea70d45ea9913a38c695" exitCode=0 Oct 30 00:34:14.000724 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.000696 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef097b1-f2wkl" event={"ID":"3665be1a-5bbd-49aa-b414-8d168419bc71","Type":"ContainerDied","Data":"76a93fc95f464804448af3c16a6ffcec1dedc2d80aaaea70d45ea9913a38c695"} Oct 30 00:34:14.011053 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.009015 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:14.011653 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.009060 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:14.032867 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.032814 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:14.106982 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.106933 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:14.109983 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.109937 5001 generic.go:356] "Generic (PLEG): container finished" podID="f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d" containerID="7db5d89a112e12eba811e2bbf825c08632fd0e712d7f6c943a219811976d0e80" exitCode=0 Oct 30 00:34:14.110418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.110391 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18e88b3-26p8b" event={"ID":"f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d","Type":"ContainerDied","Data":"7db5d89a112e12eba811e2bbf825c08632fd0e712d7f6c943a219811976d0e80"} Oct 30 00:34:14.118822 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.118790 5001 generic.go:356] "Generic (PLEG): container finished" podID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerID="693cf671fd5479bf07b7de17356b2f2d2e8e643f67e2b1d405ef4677a1af11ce" exitCode=0 Oct 30 00:34:14.120179 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.120152 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" event={"ID":"44fc62e1-bb2f-492a-af37-f412726dfb7f","Type":"ContainerDied","Data":"693cf671fd5479bf07b7de17356b2f2d2e8e643f67e2b1d405ef4677a1af11ce"} Oct 30 00:34:14.392932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.391057 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:34:14.444941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.438528 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:14.474942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.474721 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:14.491745 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.491717 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:34:14.493250 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.493220 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:34:14.699406 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.699375 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:34:14.800566 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.800505 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44fc62e1-bb2f-492a-af37-f412726dfb7f-home\" (UniqueName: \"kubernetes.io/empty-dir/44fc62e1-bb2f-492a-af37-f412726dfb7f-home\") pod \"44fc62e1-bb2f-492a-af37-f412726dfb7f\" (UID: \"44fc62e1-bb2f-492a-af37-f412726dfb7f\") " Oct 30 00:34:14.800775 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.800645 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44fc62e1-bb2f-492a-af37-f412726dfb7f-kube-api-access-cfp94\" (UniqueName: \"kubernetes.io/projected/44fc62e1-bb2f-492a-af37-f412726dfb7f-kube-api-access-cfp94\") pod \"44fc62e1-bb2f-492a-af37-f412726dfb7f\" (UID: \"44fc62e1-bb2f-492a-af37-f412726dfb7f\") " Oct 30 00:34:14.800775 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.800732 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44fc62e1-bb2f-492a-af37-f412726dfb7f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/44fc62e1-bb2f-492a-af37-f412726dfb7f-dshm\") pod \"44fc62e1-bb2f-492a-af37-f412726dfb7f\" (UID: \"44fc62e1-bb2f-492a-af37-f412726dfb7f\") " Oct 30 00:34:14.800967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.800843 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44fc62e1-bb2f-492a-af37-f412726dfb7f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44fc62e1-bb2f-492a-af37-f412726dfb7f-model-cache\") pod \"44fc62e1-bb2f-492a-af37-f412726dfb7f\" (UID: \"44fc62e1-bb2f-492a-af37-f412726dfb7f\") " Oct 30 00:34:14.801047 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.800992 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44fc62e1-bb2f-492a-af37-f412726dfb7f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44fc62e1-bb2f-492a-af37-f412726dfb7f-tmp-dir\") pod \"44fc62e1-bb2f-492a-af37-f412726dfb7f\" (UID: \"44fc62e1-bb2f-492a-af37-f412726dfb7f\") " Oct 30 00:34:14.801047 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.801030 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44fc62e1-bb2f-492a-af37-f412726dfb7f-tls-certs\" (UniqueName: \"kubernetes.io/secret/44fc62e1-bb2f-492a-af37-f412726dfb7f-tls-certs\") pod \"44fc62e1-bb2f-492a-af37-f412726dfb7f\" (UID: \"44fc62e1-bb2f-492a-af37-f412726dfb7f\") " Oct 30 00:34:14.801624 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.801598 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44fc62e1-bb2f-492a-af37-f412726dfb7f-home" pod "44fc62e1-bb2f-492a-af37-f412726dfb7f" (UID: "44fc62e1-bb2f-492a-af37-f412726dfb7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:14.801783 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.801760 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:34:14.801867 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.801822 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44fc62e1-bb2f-492a-af37-f412726dfb7f-model-cache" pod "44fc62e1-bb2f-492a-af37-f412726dfb7f" (UID: "44fc62e1-bb2f-492a-af37-f412726dfb7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:14.802487 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.802460 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44fc62e1-bb2f-492a-af37-f412726dfb7f-tmp-dir" pod "44fc62e1-bb2f-492a-af37-f412726dfb7f" (UID: "44fc62e1-bb2f-492a-af37-f412726dfb7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:14.803229 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.803202 5001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44fc62e1-bb2f-492a-af37-f412726dfb7f-model-cache\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:14.803334 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.803232 5001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44fc62e1-bb2f-492a-af37-f412726dfb7f-tmp-dir\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:14.803334 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.803244 5001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44fc62e1-bb2f-492a-af37-f412726dfb7f-home\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:14.805963 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.805945 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44fc62e1-bb2f-492a-af37-f412726dfb7f-dshm" pod "44fc62e1-bb2f-492a-af37-f412726dfb7f" (UID: "44fc62e1-bb2f-492a-af37-f412726dfb7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:14.831717 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.828075 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44fc62e1-bb2f-492a-af37-f412726dfb7f-tls-certs" pod "44fc62e1-bb2f-492a-af37-f412726dfb7f" (UID: "44fc62e1-bb2f-492a-af37-f412726dfb7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:34:14.834967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.832129 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44fc62e1-bb2f-492a-af37-f412726dfb7f-kube-api-access-cfp94" pod "44fc62e1-bb2f-492a-af37-f412726dfb7f" (UID: "44fc62e1-bb2f-492a-af37-f412726dfb7f"). InnerVolumeSpecName "kube-api-access-cfp94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:14.906556 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.906526 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfp94\" (UniqueName: \"kubernetes.io/projected/44fc62e1-bb2f-492a-af37-f412726dfb7f-kube-api-access-cfp94\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:14.906789 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.906773 5001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44fc62e1-bb2f-492a-af37-f412726dfb7f-dshm\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:14.906888 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:14.906877 5001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44fc62e1-bb2f-492a-af37-f412726dfb7f-tls-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:15.169366 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:15.169336 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" Oct 30 00:34:15.169834 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:15.169343 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" event={"ID":"44fc62e1-bb2f-492a-af37-f412726dfb7f","Type":"ContainerDied","Data":"f97c33f98f01a23ecab325013689ec43db9419bc9bde5876f5b9bcea28ff796d"} Oct 30 00:34:15.169834 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:15.169471 5001 scope.go:122] "RemoveContainer" containerID="693cf671fd5479bf07b7de17356b2f2d2e8e643f67e2b1d405ef4677a1af11ce" Oct 30 00:34:15.408992 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:15.399749 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Oct 30 00:34:15.413814 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:15.413781 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn"] Oct 30 00:34:15.414059 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:15.414044 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-564c8f/distinct-w3-564c8f-kserve-6756c6b986-x6tnn"] Oct 30 00:34:15.674597 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:15.674530 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef097b1-f2wkl" Oct 30 00:34:15.677456 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:15.677434 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18e88b3-26p8b" Oct 30 00:34:15.802566 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:15.802526 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d-kube-api-access-8pr28\" (UniqueName: \"kubernetes.io/projected/f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d-kube-api-access-8pr28\") pod \"f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d\" (UID: \"f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d\") " Oct 30 00:34:15.802761 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:15.802625 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d-maas-api-openshift-ca\") pod \"f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d\" (UID: \"f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d\") " Oct 30 00:34:15.802761 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:15.802657 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d-maas-api-xks-ca\") pod \"f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d\" (UID: \"f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d\") " Oct 30 00:34:15.802761 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:15.802715 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3665be1a-5bbd-49aa-b414-8d168419bc71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3665be1a-5bbd-49aa-b414-8d168419bc71-maas-api-xks-ca\") pod \"3665be1a-5bbd-49aa-b414-8d168419bc71\" (UID: \"3665be1a-5bbd-49aa-b414-8d168419bc71\") " Oct 30 00:34:15.803024 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:15.802825 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3665be1a-5bbd-49aa-b414-8d168419bc71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3665be1a-5bbd-49aa-b414-8d168419bc71-maas-api-openshift-ca\") pod \"3665be1a-5bbd-49aa-b414-8d168419bc71\" (UID: \"3665be1a-5bbd-49aa-b414-8d168419bc71\") " Oct 30 00:34:15.803096 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:15.803032 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3665be1a-5bbd-49aa-b414-8d168419bc71-kube-api-access-4tt7l\" (UniqueName: \"kubernetes.io/projected/3665be1a-5bbd-49aa-b414-8d168419bc71-kube-api-access-4tt7l\") pod \"3665be1a-5bbd-49aa-b414-8d168419bc71\" (UID: \"3665be1a-5bbd-49aa-b414-8d168419bc71\") " Oct 30 00:34:15.803298 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:15.803274 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3665be1a-5bbd-49aa-b414-8d168419bc71-maas-api-openshift-ca" pod "3665be1a-5bbd-49aa-b414-8d168419bc71" (UID: "3665be1a-5bbd-49aa-b414-8d168419bc71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:34:15.803822 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:15.803796 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3665be1a-5bbd-49aa-b414-8d168419bc71-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:15.803960 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:15.803851 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d-maas-api-openshift-ca" pod "f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d" (UID: "f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:34:15.838937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:15.817125 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3665be1a-5bbd-49aa-b414-8d168419bc71-kube-api-access-4tt7l" pod "3665be1a-5bbd-49aa-b414-8d168419bc71" (UID: "3665be1a-5bbd-49aa-b414-8d168419bc71"). InnerVolumeSpecName "kube-api-access-4tt7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:15.841859 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:15.840940 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d-maas-api-xks-ca" pod "f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d" (UID: "f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:34:15.850963 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:15.843049 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3665be1a-5bbd-49aa-b414-8d168419bc71-maas-api-xks-ca" pod "3665be1a-5bbd-49aa-b414-8d168419bc71" (UID: "3665be1a-5bbd-49aa-b414-8d168419bc71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:34:15.855462 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:15.855431 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d-kube-api-access-8pr28" pod "f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d" (UID: "f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d"). InnerVolumeSpecName "kube-api-access-8pr28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:15.904774 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:15.904745 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pr28\" (UniqueName: \"kubernetes.io/projected/f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d-kube-api-access-8pr28\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:15.904774 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:15.904774 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:15.904774 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:15.904785 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:15.905027 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:15.904795 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3665be1a-5bbd-49aa-b414-8d168419bc71-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:15.905027 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:15.904805 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tt7l\" (UniqueName: \"kubernetes.io/projected/3665be1a-5bbd-49aa-b414-8d168419bc71-kube-api-access-4tt7l\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:16.181899 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:16.181868 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18e88b3-26p8b" Oct 30 00:34:16.181899 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:16.181893 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18e88b3-26p8b" event={"ID":"f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d","Type":"ContainerDied","Data":"407905dd4cda5d802f4e01fda7b73bcc30b7a9d1a30c273688fa845b7f28f7de"} Oct 30 00:34:16.182390 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:16.181958 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="407905dd4cda5d802f4e01fda7b73bcc30b7a9d1a30c273688fa845b7f28f7de" Oct 30 00:34:16.184845 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:16.184823 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef097b1-f2wkl" Oct 30 00:34:16.184845 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:16.184838 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef097b1-f2wkl" event={"ID":"3665be1a-5bbd-49aa-b414-8d168419bc71","Type":"ContainerDied","Data":"06a74176a0af455f4a7ae0724ddae2a25d4320b6e71695aa38f00e4137d81d4e"} Oct 30 00:34:16.185077 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:16.184880 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06a74176a0af455f4a7ae0724ddae2a25d4320b6e71695aa38f00e4137d81d4e" Oct 30 00:34:16.982978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:16.982448 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" path="/var/lib/kubelet/pods/44fc62e1-bb2f-492a-af37-f412726dfb7f/volumes" Oct 30 00:34:17.144944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:17.144904 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f1268c-7745d464f5-hf7k2" Oct 30 00:34:17.359809 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:34:17.359764 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:34:21.299690 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:21.299663 5001 generic.go:356] "Generic (PLEG): container finished" podID="aadaf6fb-2702-4c04-88c9-cbb181049d68" containerID="35460dd73d82bac9413321658766b2cade487258b6d5c905472184b20bbed8d6" exitCode=0 Oct 30 00:34:21.300169 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:21.299733 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" event={"ID":"aadaf6fb-2702-4c04-88c9-cbb181049d68","Type":"ContainerDied","Data":"35460dd73d82bac9413321658766b2cade487258b6d5c905472184b20bbed8d6"} Oct 30 00:34:21.300265 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:21.300247 5001 scope.go:122] "RemoveContainer" containerID="35460dd73d82bac9413321658766b2cade487258b6d5c905472184b20bbed8d6" Oct 30 00:34:21.302590 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:21.302552 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:34:21.302775 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:21.302649 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:34:21.302775 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:21.302700 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:34:22.362069 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:22.362031 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" event={"ID":"aadaf6fb-2702-4c04-88c9-cbb181049d68","Type":"ContainerStarted","Data":"eeaebb279140456c722be18222cfe34fe4c8f4092ef689e75a754e377382c219"} Oct 30 00:34:23.141843 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:23.141812 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:34:23.142248 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:23.142224 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 00:34:23.142420 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:23.142395 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" podUID="aadaf6fb-2702-4c04-88c9-cbb181049d68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 00:34:23.345340 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:23.345312 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-564c8f-57dbdd8c55-q6gz2" Oct 30 00:34:23.373717 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:23.373690 5001 generic.go:356] "Generic (PLEG): container finished" podID="85331496-4439-48d1-aca8-19dc5cf91bae" containerID="3cbe2ffbe68f15f32a979205b3d35ab61a6e5e80e49806a3543017ac7ec0895e" exitCode=0 Oct 30 00:34:23.374150 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:23.373766 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" event={"ID":"85331496-4439-48d1-aca8-19dc5cf91bae","Type":"ContainerDied","Data":"3cbe2ffbe68f15f32a979205b3d35ab61a6e5e80e49806a3543017ac7ec0895e"} Oct 30 00:34:23.374310 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:23.374280 5001 scope.go:122] "RemoveContainer" containerID="3cbe2ffbe68f15f32a979205b3d35ab61a6e5e80e49806a3543017ac7ec0895e" Oct 30 00:34:23.375399 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:23.375375 5001 generic.go:356] "Generic (PLEG): container finished" podID="2077ddd9-c805-453e-8611-fc8b90b3b3c1" containerID="96a3d3e90bb2f7db6d7bd8518d9bd7080b09a4f9f963d7681f36980417f8d102" exitCode=0 Oct 30 00:34:23.375835 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:23.375457 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-564c8f-57dbdd8c55-q6gz2" Oct 30 00:34:23.375835 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:23.375494 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-564c8f-57dbdd8c55-q6gz2" event={"ID":"2077ddd9-c805-453e-8611-fc8b90b3b3c1","Type":"ContainerDied","Data":"96a3d3e90bb2f7db6d7bd8518d9bd7080b09a4f9f963d7681f36980417f8d102"} Oct 30 00:34:23.375835 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:23.375553 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-564c8f-57dbdd8c55-q6gz2" event={"ID":"2077ddd9-c805-453e-8611-fc8b90b3b3c1","Type":"ContainerDied","Data":"e3b5c8be47bcabd837b7a4a187bd2bf5b3222059e24d4cb62a60963019c76ace"} Oct 30 00:34:23.375835 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:23.375567 5001 scope.go:122] "RemoveContainer" containerID="96a3d3e90bb2f7db6d7bd8518d9bd7080b09a4f9f963d7681f36980417f8d102" Oct 30 00:34:23.376232 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:23.376155 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:34:23.376336 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:23.376238 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:34:23.376336 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:23.376279 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:34:23.397663 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:23.397631 5001 scope.go:122] "RemoveContainer" containerID="96a3d3e90bb2f7db6d7bd8518d9bd7080b09a4f9f963d7681f36980417f8d102" Oct 30 00:34:23.398042 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:34:23.398004 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96a3d3e90bb2f7db6d7bd8518d9bd7080b09a4f9f963d7681f36980417f8d102\": container with ID starting with 96a3d3e90bb2f7db6d7bd8518d9bd7080b09a4f9f963d7681f36980417f8d102 not found: ID does not exist" containerID="96a3d3e90bb2f7db6d7bd8518d9bd7080b09a4f9f963d7681f36980417f8d102" Oct 30 00:34:23.398157 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:23.398055 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96a3d3e90bb2f7db6d7bd8518d9bd7080b09a4f9f963d7681f36980417f8d102"} err="failed to get container status \"96a3d3e90bb2f7db6d7bd8518d9bd7080b09a4f9f963d7681f36980417f8d102\": rpc error: code = NotFound desc = could not find container \"96a3d3e90bb2f7db6d7bd8518d9bd7080b09a4f9f963d7681f36980417f8d102\": container with ID starting with 96a3d3e90bb2f7db6d7bd8518d9bd7080b09a4f9f963d7681f36980417f8d102 not found: ID does not exist" Oct 30 00:34:23.451984 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:23.451935 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2077ddd9-c805-453e-8611-fc8b90b3b3c1-kube-api-access-ks9x7\" (UniqueName: \"kubernetes.io/projected/2077ddd9-c805-453e-8611-fc8b90b3b3c1-kube-api-access-ks9x7\") pod \"2077ddd9-c805-453e-8611-fc8b90b3b3c1\" (UID: \"2077ddd9-c805-453e-8611-fc8b90b3b3c1\") " Oct 30 00:34:23.452296 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:23.452270 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2077ddd9-c805-453e-8611-fc8b90b3b3c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2077ddd9-c805-453e-8611-fc8b90b3b3c1-plugins-config-volume\") pod \"2077ddd9-c805-453e-8611-fc8b90b3b3c1\" (UID: \"2077ddd9-c805-453e-8611-fc8b90b3b3c1\") " Oct 30 00:34:23.455220 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:23.453459 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2077ddd9-c805-453e-8611-fc8b90b3b3c1-plugins-config-volume" pod "2077ddd9-c805-453e-8611-fc8b90b3b3c1" (UID: "2077ddd9-c805-453e-8611-fc8b90b3b3c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:34:23.455220 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:23.454900 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2077ddd9-c805-453e-8611-fc8b90b3b3c1-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:23.478086 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:23.478055 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2077ddd9-c805-453e-8611-fc8b90b3b3c1-kube-api-access-ks9x7" pod "2077ddd9-c805-453e-8611-fc8b90b3b3c1" (UID: "2077ddd9-c805-453e-8611-fc8b90b3b3c1"). InnerVolumeSpecName "kube-api-access-ks9x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:23.561024 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:23.560980 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks9x7\" (UniqueName: \"kubernetes.io/projected/2077ddd9-c805-453e-8611-fc8b90b3b3c1-kube-api-access-ks9x7\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:23.743459 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:23.743423 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-564c8f-57dbdd8c55-q6gz2"] Oct 30 00:34:23.749156 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:23.749126 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-564c8f-57dbdd8c55-q6gz2"] Oct 30 00:34:24.138982 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:24.137355 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 00:34:24.138982 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:24.137432 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" podUID="aadaf6fb-2702-4c04-88c9-cbb181049d68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 00:34:24.157955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:24.157890 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-564c8f-57dbdd8c55-q6gz2" podUID="2077ddd9-c805-453e-8611-fc8b90b3b3c1" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.76:50052\" within 1s: context deadline exceeded" Oct 30 00:34:24.158145 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:34:24.158038 5001 logging.go:55] [core] [Channel #3317 SubChannel #3318]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.76:50052", ServerName: "10.132.0.76:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.76:50052: operation was canceled" Oct 30 00:34:24.458987 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:24.458166 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" event={"ID":"85331496-4439-48d1-aca8-19dc5cf91bae","Type":"ContainerStarted","Data":"472a381bccf53ec6b6af885f3d3685480bf630ac169e07b4eb8cc3696c5b6d91"} Oct 30 00:34:25.020948 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:25.017707 5001 patch_prober.go:29] interesting pod/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body= Oct 30 00:34:25.020948 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:25.017783 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" podUID="85331496-4439-48d1-aca8-19dc5cf91bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" Oct 30 00:34:25.127481 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:25.127439 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2077ddd9-c805-453e-8611-fc8b90b3b3c1" path="/var/lib/kubelet/pods/2077ddd9-c805-453e-8611-fc8b90b3b3c1/volumes" Oct 30 00:34:25.128290 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:25.128267 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-564c8f-wgm5x"] Oct 30 00:34:25.128477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:25.128463 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:34:25.140878 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:25.139310 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 00:34:25.140878 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:25.139369 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" podUID="aadaf6fb-2702-4c04-88c9-cbb181049d68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 00:34:25.154954 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:25.145264 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-564c8f-wgm5x"] Oct 30 00:34:25.956877 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:25.956841 5001 patch_prober.go:29] interesting pod/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body= Oct 30 00:34:25.957510 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:25.957475 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" podUID="85331496-4439-48d1-aca8-19dc5cf91bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" Oct 30 00:34:26.135211 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:26.135177 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 00:34:26.135500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:26.135470 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" podUID="aadaf6fb-2702-4c04-88c9-cbb181049d68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 00:34:26.974938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:26.954434 5001 patch_prober.go:29] interesting pod/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body= Oct 30 00:34:26.974938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:26.954533 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" podUID="85331496-4439-48d1-aca8-19dc5cf91bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" Oct 30 00:34:27.046006 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:27.045954 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 00:34:27.139233 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:27.137630 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 00:34:27.139233 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:27.137693 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" podUID="aadaf6fb-2702-4c04-88c9-cbb181049d68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 00:34:27.220936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:27.215898 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf200711-0050-4fae-8fe0-385b6e39dbe2" path="/var/lib/kubelet/pods/cf200711-0050-4fae-8fe0-385b6e39dbe2/volumes" Oct 30 00:34:27.459938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:27.457743 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:27.822075 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:27.822040 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:34:27.844100 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:27.844055 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:27.953376 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:27.953334 5001 patch_prober.go:29] interesting pod/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body= Oct 30 00:34:27.953652 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:27.953623 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" podUID="85331496-4439-48d1-aca8-19dc5cf91bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" Oct 30 00:34:28.057989 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:28.057903 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:28.059125 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:28.059101 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:34:28.138366 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:28.137329 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 00:34:28.138366 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:28.137390 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" podUID="aadaf6fb-2702-4c04-88c9-cbb181049d68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 00:34:28.458253 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:28.458157 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:28.459031 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:28.459007 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:34:28.950937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:28.950435 5001 patch_prober.go:29] interesting pod/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body= Oct 30 00:34:28.950937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:28.950505 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" podUID="85331496-4439-48d1-aca8-19dc5cf91bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" Oct 30 00:34:29.113153 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.113092 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:29.189938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.162852 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 00:34:29.189938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.162929 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" podUID="aadaf6fb-2702-4c04-88c9-cbb181049d68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 00:34:29.364186 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.364139 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:29.722310 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.722219 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv"] Oct 30 00:34:29.794923 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.794876 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b492299d-9c08-491b-bcd8-3d4ae73f5522" containerName="revoke-keys" Oct 30 00:34:29.795176 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.795156 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b492299d-9c08-491b-bcd8-3d4ae73f5522" containerName="revoke-keys" Oct 30 00:34:29.795346 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.795332 5001 cpu_manager.go:414] "containerMap: removing container" podUID="2a4eee95-140b-45a8-a9a8-0c5b2be11e65" containerName="revoke-keys" Oct 30 00:34:29.804009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.803980 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a4eee95-140b-45a8-a9a8-0c5b2be11e65" containerName="revoke-keys" Oct 30 00:34:29.804267 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.804245 5001 cpu_manager.go:414] "containerMap: removing container" podUID="3665be1a-5bbd-49aa-b414-8d168419bc71" containerName="revoke-keys" Oct 30 00:34:29.804378 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.804365 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3665be1a-5bbd-49aa-b414-8d168419bc71" containerName="revoke-keys" Oct 30 00:34:29.804486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.804475 5001 cpu_manager.go:414] "containerMap: removing container" podUID="2077ddd9-c805-453e-8611-fc8b90b3b3c1" containerName="payload-pre-processing" Oct 30 00:34:29.804589 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.804577 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2077ddd9-c805-453e-8611-fc8b90b3b3c1" containerName="payload-pre-processing" Oct 30 00:34:29.804708 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.804697 5001 cpu_manager.go:414] "containerMap: removing container" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" Oct 30 00:34:29.804807 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.804795 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" Oct 30 00:34:29.805005 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.804985 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d" containerName="revoke-keys" Oct 30 00:34:29.805115 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.805103 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d" containerName="revoke-keys" Oct 30 00:34:29.805524 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.805497 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a4eee95-140b-45a8-a9a8-0c5b2be11e65" containerName="revoke-keys" Oct 30 00:34:29.805657 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.805647 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3665be1a-5bbd-49aa-b414-8d168419bc71" containerName="revoke-keys" Oct 30 00:34:29.805758 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.805748 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2077ddd9-c805-453e-8611-fc8b90b3b3c1" containerName="payload-pre-processing" Oct 30 00:34:29.805881 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.805869 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="44fc62e1-bb2f-492a-af37-f412726dfb7f" containerName="main" Oct 30 00:34:29.806051 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.806037 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d" containerName="revoke-keys" Oct 30 00:34:29.806175 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.806164 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b492299d-9c08-491b-bcd8-3d4ae73f5522" containerName="revoke-keys" Oct 30 00:34:29.820170 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.820128 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:34:29.820831 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.820810 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv"] Oct 30 00:34:29.822148 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.822126 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:34:29.823052 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.820445 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv" Oct 30 00:34:29.873544 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.873507 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-be1f39\"/\"default-dockercfg-mznkl\"" Oct 30 00:34:29.873933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.873897 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-be1f39\"/\"explicit-model-be1f39-kserve-self-signed-certs\"" Oct 30 00:34:29.874168 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.874150 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-be1f39\"/\"kube-root-ca.crt\"" Oct 30 00:34:29.874376 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.874359 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-be1f39\"/\"openshift-service-ca.crt\"" Oct 30 00:34:29.972987 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.954537 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2d9f394-c972-4b00-ba20-6bd04d977827-tls-certs\") pod \"explicit-model-be1f39-kserve-6bc4555567-8mkkv\" (UID: \"b2d9f394-c972-4b00-ba20-6bd04d977827\") " pod="ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv" Oct 30 00:34:29.973323 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.973292 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f394-c972-4b00-ba20-6bd04d977827-model-cache\") pod \"explicit-model-be1f39-kserve-6bc4555567-8mkkv\" (UID: \"b2d9f394-c972-4b00-ba20-6bd04d977827\") " pod="ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv" Oct 30 00:34:29.973542 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.973522 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l97gb\" (UniqueName: \"kubernetes.io/projected/b2d9f394-c972-4b00-ba20-6bd04d977827-kube-api-access-l97gb\") pod \"explicit-model-be1f39-kserve-6bc4555567-8mkkv\" (UID: \"b2d9f394-c972-4b00-ba20-6bd04d977827\") " pod="ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv" Oct 30 00:34:29.973866 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.973849 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f394-c972-4b00-ba20-6bd04d977827-tmp-dir\") pod \"explicit-model-be1f39-kserve-6bc4555567-8mkkv\" (UID: \"b2d9f394-c972-4b00-ba20-6bd04d977827\") " pod="ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv" Oct 30 00:34:29.974014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.973992 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f394-c972-4b00-ba20-6bd04d977827-home\") pod \"explicit-model-be1f39-kserve-6bc4555567-8mkkv\" (UID: \"b2d9f394-c972-4b00-ba20-6bd04d977827\") " pod="ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv" Oct 30 00:34:29.974262 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.974244 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f394-c972-4b00-ba20-6bd04d977827-dshm\") pod \"explicit-model-be1f39-kserve-6bc4555567-8mkkv\" (UID: \"b2d9f394-c972-4b00-ba20-6bd04d977827\") " pod="ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv" Oct 30 00:34:29.974608 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.965653 5001 patch_prober.go:29] interesting pod/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body= Oct 30 00:34:29.974772 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:29.974746 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" podUID="85331496-4439-48d1-aca8-19dc5cf91bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" Oct 30 00:34:30.275937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:30.270015 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l97gb\" (UniqueName: \"kubernetes.io/projected/b2d9f394-c972-4b00-ba20-6bd04d977827-kube-api-access-l97gb\") pod \"explicit-model-be1f39-kserve-6bc4555567-8mkkv\" (UID: \"b2d9f394-c972-4b00-ba20-6bd04d977827\") " pod="ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv" Oct 30 00:34:30.275937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:30.270170 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f394-c972-4b00-ba20-6bd04d977827-tmp-dir\") pod \"explicit-model-be1f39-kserve-6bc4555567-8mkkv\" (UID: \"b2d9f394-c972-4b00-ba20-6bd04d977827\") " pod="ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv" Oct 30 00:34:30.275937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:30.270201 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f394-c972-4b00-ba20-6bd04d977827-home\") pod \"explicit-model-be1f39-kserve-6bc4555567-8mkkv\" (UID: \"b2d9f394-c972-4b00-ba20-6bd04d977827\") " pod="ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv" Oct 30 00:34:30.275937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:30.270293 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f394-c972-4b00-ba20-6bd04d977827-dshm\") pod \"explicit-model-be1f39-kserve-6bc4555567-8mkkv\" (UID: \"b2d9f394-c972-4b00-ba20-6bd04d977827\") " pod="ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv" Oct 30 00:34:30.275937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:30.270447 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2d9f394-c972-4b00-ba20-6bd04d977827-tls-certs\") pod \"explicit-model-be1f39-kserve-6bc4555567-8mkkv\" (UID: \"b2d9f394-c972-4b00-ba20-6bd04d977827\") " pod="ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv" Oct 30 00:34:30.275937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:30.270484 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f394-c972-4b00-ba20-6bd04d977827-model-cache\") pod \"explicit-model-be1f39-kserve-6bc4555567-8mkkv\" (UID: \"b2d9f394-c972-4b00-ba20-6bd04d977827\") " pod="ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv" Oct 30 00:34:30.376450 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:30.376410 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:30.382948 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:30.377237 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:30.448945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:30.394421 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f394-c972-4b00-ba20-6bd04d977827-model-cache\") pod \"explicit-model-be1f39-kserve-6bc4555567-8mkkv\" (UID: \"b2d9f394-c972-4b00-ba20-6bd04d977827\") " pod="ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv" Oct 30 00:34:30.448945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:30.439940 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f394-c972-4b00-ba20-6bd04d977827-home\") pod \"explicit-model-be1f39-kserve-6bc4555567-8mkkv\" (UID: \"b2d9f394-c972-4b00-ba20-6bd04d977827\") " pod="ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv" Oct 30 00:34:30.448945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:30.440352 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 00:34:30.448945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:30.440401 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" podUID="aadaf6fb-2702-4c04-88c9-cbb181049d68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 00:34:30.574639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:30.574564 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f394-c972-4b00-ba20-6bd04d977827-tmp-dir\") pod \"explicit-model-be1f39-kserve-6bc4555567-8mkkv\" (UID: \"b2d9f394-c972-4b00-ba20-6bd04d977827\") " pod="ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv" Oct 30 00:34:30.632618 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:30.629580 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2d9f394-c972-4b00-ba20-6bd04d977827-tls-certs\") pod \"explicit-model-be1f39-kserve-6bc4555567-8mkkv\" (UID: \"b2d9f394-c972-4b00-ba20-6bd04d977827\") " pod="ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv" Oct 30 00:34:30.632618 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:30.632337 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:30.747529 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:30.747487 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f394-c972-4b00-ba20-6bd04d977827-dshm\") pod \"explicit-model-be1f39-kserve-6bc4555567-8mkkv\" (UID: \"b2d9f394-c972-4b00-ba20-6bd04d977827\") " pod="ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv" Oct 30 00:34:30.880422 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:30.880316 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:34:31.169992 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:31.160980 5001 patch_prober.go:29] interesting pod/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body= Oct 30 00:34:31.171828 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:31.171775 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" podUID="85331496-4439-48d1-aca8-19dc5cf91bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" Oct 30 00:34:31.191902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:31.191867 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 00:34:31.192175 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:31.192144 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" podUID="aadaf6fb-2702-4c04-88c9-cbb181049d68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 00:34:31.293899 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:31.293866 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l97gb\" (UniqueName: \"kubernetes.io/projected/b2d9f394-c972-4b00-ba20-6bd04d977827-kube-api-access-l97gb\") pod \"explicit-model-be1f39-kserve-6bc4555567-8mkkv\" (UID: \"b2d9f394-c972-4b00-ba20-6bd04d977827\") " pod="ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv" Oct 30 00:34:31.477718 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:31.477601 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv" Oct 30 00:34:31.481564 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:31.481533 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:34:31.953350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:31.953311 5001 patch_prober.go:29] interesting pod/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body= Oct 30 00:34:31.953633 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:31.953604 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" podUID="85331496-4439-48d1-aca8-19dc5cf91bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" Oct 30 00:34:32.308943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:32.308771 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:34:32.331620 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:32.331582 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:34:32.385087 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:32.385019 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv"] Oct 30 00:34:32.572681 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:32.572589 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6"] Oct 30 00:34:32.572860 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:32.572821 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" podUID="b2cdf918-8248-4079-9823-830a84e927bf" containerName="istio-proxy" containerID="cri-o://69e51c74081b349c18d147927e847f86aedaf9f4774f2b7098fa342db180dc58" gracePeriod=30 Oct 30 00:34:32.585536 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:32.585489 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv" event={"ID":"b2d9f394-c972-4b00-ba20-6bd04d977827","Type":"ContainerStarted","Data":"7623b40e98b1d1dd0ecadaa3bba1094342af29bbfa3ec7e4e1bd83e68e113451"} Oct 30 00:34:32.759001 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:32.758965 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8"] Oct 30 00:34:32.774782 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:32.774740 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" podUID="9f93e25c-1a99-48c6-8a27-05e8ea723a6d" containerName="istio-proxy" containerID="cri-o://ce14b911b882bfb0ef52d120c36ffd835e196f7d259dc09b7991c7a79b099001" gracePeriod=30 Oct 30 00:34:32.890127 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:32.890032 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:34:32.890424 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:32.890370 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" podUID="b2cdf918-8248-4079-9823-830a84e927bf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:34:32.955116 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:32.955075 5001 patch_prober.go:29] interesting pod/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body= Oct 30 00:34:32.955392 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:32.955363 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" podUID="85331496-4439-48d1-aca8-19dc5cf91bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" Oct 30 00:34:33.048640 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:33.048604 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s"] Oct 30 00:34:33.050500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:33.050463 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" podUID="f549cc40-a1ae-43aa-825f-11d538b9fa71" containerName="istio-proxy" containerID="cri-o://d93b8a9f02863d14cd884a190ee8aebf784d7e11d8b488b2342554bccacab816" gracePeriod=30 Oct 30 00:34:33.284842 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:33.284800 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:34:33.285014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:33.284887 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" podUID="f549cc40-a1ae-43aa-825f-11d538b9fa71" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:34:33.612603 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:33.612563 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv" event={"ID":"b2d9f394-c972-4b00-ba20-6bd04d977827","Type":"ContainerStarted","Data":"f4b3c20a953dea467b583c9e56500c2b10276b746805331b09820c9009ccbd14"} Oct 30 00:34:33.660453 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:33.660397 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv" podStartSLOduration=5.660384412 podStartE2EDuration="5.660384412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:33.659286887 +0000 UTC m=+1788.864788081" watchObservedRunningTime="2026-10-30 00:34:33.660384412 +0000 UTC m=+1788.865885590" Oct 30 00:34:33.949716 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:33.949630 5001 patch_prober.go:29] interesting pod/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body= Oct 30 00:34:33.949716 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:33.949700 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" podUID="85331496-4439-48d1-aca8-19dc5cf91bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" Oct 30 00:34:34.033477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:34.033435 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-422c9df5-v25fr"] Oct 30 00:34:34.040962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:34.040895 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-422c9df5-v25fr"] Oct 30 00:34:34.483627 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:34.483587 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv" Oct 30 00:34:35.211941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:35.195403 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9aa57dc-4f61-4001-8573-9cd8ee69246f" path="/var/lib/kubelet/pods/a9aa57dc-4f61-4001-8573-9cd8ee69246f/volumes" Oct 30 00:34:35.211941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:35.196216 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:34:35.214967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:35.214942 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:34:35.646065 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:35.646026 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1c718-mjsnx"] Oct 30 00:34:35.657991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:35.657960 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1c718-mjsnx"] Oct 30 00:34:36.921656 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:36.921614 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89e9208c-ef2c-483d-a9bb-666416fd9076" path="/var/lib/kubelet/pods/89e9208c-ef2c-483d-a9bb-666416fd9076/volumes" Oct 30 00:34:37.730138 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:37.730108 5001 generic.go:356] "Generic (PLEG): container finished" podID="b2cdf918-8248-4079-9823-830a84e927bf" containerID="69e51c74081b349c18d147927e847f86aedaf9f4774f2b7098fa342db180dc58" exitCode=0 Oct 30 00:34:37.730335 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:37.730195 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" event={"ID":"b2cdf918-8248-4079-9823-830a84e927bf","Type":"ContainerDied","Data":"69e51c74081b349c18d147927e847f86aedaf9f4774f2b7098fa342db180dc58"} Oct 30 00:34:37.951605 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:37.951558 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:34:38.100215 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.100104 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-workload-certs\") pod \"b2cdf918-8248-4079-9823-830a84e927bf\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " Oct 30 00:34:38.100215 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.100190 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2cdf918-8248-4079-9823-830a84e927bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2cdf918-8248-4079-9823-830a84e927bf-istiod-ca-cert\") pod \"b2cdf918-8248-4079-9823-830a84e927bf\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " Oct 30 00:34:38.100380 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.100269 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-workload-socket\") pod \"b2cdf918-8248-4079-9823-830a84e927bf\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " Oct 30 00:34:38.100380 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.100351 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-istio-envoy\") pod \"b2cdf918-8248-4079-9823-830a84e927bf\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " Oct 30 00:34:38.100488 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.100401 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2cdf918-8248-4079-9823-830a84e927bf-istio-token\" (UniqueName: \"kubernetes.io/projected/b2cdf918-8248-4079-9823-830a84e927bf-istio-token\") pod \"b2cdf918-8248-4079-9823-830a84e927bf\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " Oct 30 00:34:38.100558 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.100496 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2cdf918-8248-4079-9823-830a84e927bf-kube-api-access-sjb4l\" (UniqueName: \"kubernetes.io/projected/b2cdf918-8248-4079-9823-830a84e927bf-kube-api-access-sjb4l\") pod \"b2cdf918-8248-4079-9823-830a84e927bf\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " Oct 30 00:34:38.100620 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.100571 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-workload-socket" pod "b2cdf918-8248-4079-9823-830a84e927bf" (UID: "b2cdf918-8248-4079-9823-830a84e927bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.100703 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.100682 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2cdf918-8248-4079-9823-830a84e927bf-istiod-ca-cert" pod "b2cdf918-8248-4079-9823-830a84e927bf" (UID: "b2cdf918-8248-4079-9823-830a84e927bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:34:38.100771 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.100703 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-istio-data\") pod \"b2cdf918-8248-4079-9823-830a84e927bf\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " Oct 30 00:34:38.100838 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.100818 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2cdf918-8248-4079-9823-830a84e927bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2cdf918-8248-4079-9823-830a84e927bf-istio-podinfo\") pod \"b2cdf918-8248-4079-9823-830a84e927bf\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " Oct 30 00:34:38.100902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.100880 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-credential-socket\") pod \"b2cdf918-8248-4079-9823-830a84e927bf\" (UID: \"b2cdf918-8248-4079-9823-830a84e927bf\") " Oct 30 00:34:38.100988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.100900 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-workload-certs" pod "b2cdf918-8248-4079-9823-830a84e927bf" (UID: "b2cdf918-8248-4079-9823-830a84e927bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.101320 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.101295 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-istio-data" pod "b2cdf918-8248-4079-9823-830a84e927bf" (UID: "b2cdf918-8248-4079-9823-830a84e927bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.101320 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.101311 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-credential-socket" pod "b2cdf918-8248-4079-9823-830a84e927bf" (UID: "b2cdf918-8248-4079-9823-830a84e927bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.101778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.101756 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.101820 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.101785 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.101820 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.101801 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2cdf918-8248-4079-9823-830a84e927bf-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.101820 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.101813 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.101930 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.101828 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.103320 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.103292 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2cdf918-8248-4079-9823-830a84e927bf-istio-token" pod "b2cdf918-8248-4079-9823-830a84e927bf" (UID: "b2cdf918-8248-4079-9823-830a84e927bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:38.103511 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.103492 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2cdf918-8248-4079-9823-830a84e927bf-kube-api-access-sjb4l" pod "b2cdf918-8248-4079-9823-830a84e927bf" (UID: "b2cdf918-8248-4079-9823-830a84e927bf"). InnerVolumeSpecName "kube-api-access-sjb4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:38.103828 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.103809 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-istio-envoy" pod "b2cdf918-8248-4079-9823-830a84e927bf" (UID: "b2cdf918-8248-4079-9823-830a84e927bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.103895 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.103824 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2cdf918-8248-4079-9823-830a84e927bf-istio-podinfo" pod "b2cdf918-8248-4079-9823-830a84e927bf" (UID: "b2cdf918-8248-4079-9823-830a84e927bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:34:38.150764 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.150690 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:34:38.209015 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.206935 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2cdf918-8248-4079-9823-830a84e927bf-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.209015 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.206973 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2cdf918-8248-4079-9823-830a84e927bf-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.209015 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.206987 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2cdf918-8248-4079-9823-830a84e927bf-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.209015 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.207000 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjb4l\" (UniqueName: \"kubernetes.io/projected/b2cdf918-8248-4079-9823-830a84e927bf-kube-api-access-sjb4l\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.284320 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.284280 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:34:38.307608 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.307569 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-credential-socket\") pod \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " Oct 30 00:34:38.307800 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.307678 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istio-token\" (UniqueName: \"kubernetes.io/projected/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istio-token\") pod \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " Oct 30 00:34:38.307800 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.307728 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istiod-ca-cert\") pod \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " Oct 30 00:34:38.307800 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.307760 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istio-data\") pod \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " Oct 30 00:34:38.307990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.307811 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-workload-certs\") pod \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " Oct 30 00:34:38.308036 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.307995 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-kube-api-access-tql2w\" (UniqueName: \"kubernetes.io/projected/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-kube-api-access-tql2w\") pod \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " Oct 30 00:34:38.308036 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.308026 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-workload-socket\") pod \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " Oct 30 00:34:38.308145 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.308129 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istio-envoy\") pod \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " Oct 30 00:34:38.308289 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.308274 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istio-podinfo\") pod \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\" (UID: \"9f93e25c-1a99-48c6-8a27-05e8ea723a6d\") " Oct 30 00:34:38.309121 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.309089 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istio-data" pod "9f93e25c-1a99-48c6-8a27-05e8ea723a6d" (UID: "9f93e25c-1a99-48c6-8a27-05e8ea723a6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.309427 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.309399 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-workload-socket" pod "9f93e25c-1a99-48c6-8a27-05e8ea723a6d" (UID: "9f93e25c-1a99-48c6-8a27-05e8ea723a6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.309508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.309431 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-credential-socket" pod "9f93e25c-1a99-48c6-8a27-05e8ea723a6d" (UID: "9f93e25c-1a99-48c6-8a27-05e8ea723a6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.309508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.309455 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-workload-certs" pod "9f93e25c-1a99-48c6-8a27-05e8ea723a6d" (UID: "9f93e25c-1a99-48c6-8a27-05e8ea723a6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.309634 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.309584 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istiod-ca-cert" pod "9f93e25c-1a99-48c6-8a27-05e8ea723a6d" (UID: "9f93e25c-1a99-48c6-8a27-05e8ea723a6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:34:38.312000 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.311971 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istio-envoy" pod "9f93e25c-1a99-48c6-8a27-05e8ea723a6d" (UID: "9f93e25c-1a99-48c6-8a27-05e8ea723a6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.312094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.312029 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-kube-api-access-tql2w" pod "9f93e25c-1a99-48c6-8a27-05e8ea723a6d" (UID: "9f93e25c-1a99-48c6-8a27-05e8ea723a6d"). InnerVolumeSpecName "kube-api-access-tql2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:38.312273 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.312246 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istio-podinfo" pod "9f93e25c-1a99-48c6-8a27-05e8ea723a6d" (UID: "9f93e25c-1a99-48c6-8a27-05e8ea723a6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:34:38.312444 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.312425 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istio-token" pod "9f93e25c-1a99-48c6-8a27-05e8ea723a6d" (UID: "9f93e25c-1a99-48c6-8a27-05e8ea723a6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:38.422855 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.422759 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-workload-socket\") pod \"f549cc40-a1ae-43aa-825f-11d538b9fa71\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " Oct 30 00:34:38.423051 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.422906 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-istio-envoy\") pod \"f549cc40-a1ae-43aa-825f-11d538b9fa71\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " Oct 30 00:34:38.423051 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.422962 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f549cc40-a1ae-43aa-825f-11d538b9fa71-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f549cc40-a1ae-43aa-825f-11d538b9fa71-istiod-ca-cert\") pod \"f549cc40-a1ae-43aa-825f-11d538b9fa71\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " Oct 30 00:34:38.423051 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.423018 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-workload-certs\") pod \"f549cc40-a1ae-43aa-825f-11d538b9fa71\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " Oct 30 00:34:38.423051 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.423029 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-workload-socket" pod "f549cc40-a1ae-43aa-825f-11d538b9fa71" (UID: "f549cc40-a1ae-43aa-825f-11d538b9fa71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.423290 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.423069 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f549cc40-a1ae-43aa-825f-11d538b9fa71-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f549cc40-a1ae-43aa-825f-11d538b9fa71-istio-podinfo\") pod \"f549cc40-a1ae-43aa-825f-11d538b9fa71\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " Oct 30 00:34:38.423290 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.423160 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f549cc40-a1ae-43aa-825f-11d538b9fa71-istio-token\" (UniqueName: \"kubernetes.io/projected/f549cc40-a1ae-43aa-825f-11d538b9fa71-istio-token\") pod \"f549cc40-a1ae-43aa-825f-11d538b9fa71\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " Oct 30 00:34:38.423290 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.423202 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-credential-socket\") pod \"f549cc40-a1ae-43aa-825f-11d538b9fa71\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " Oct 30 00:34:38.423290 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.423279 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-istio-data\") pod \"f549cc40-a1ae-43aa-825f-11d538b9fa71\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " Oct 30 00:34:38.423542 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.423332 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f549cc40-a1ae-43aa-825f-11d538b9fa71-kube-api-access-gx6ct\" (UniqueName: \"kubernetes.io/projected/f549cc40-a1ae-43aa-825f-11d538b9fa71-kube-api-access-gx6ct\") pod \"f549cc40-a1ae-43aa-825f-11d538b9fa71\" (UID: \"f549cc40-a1ae-43aa-825f-11d538b9fa71\") " Oct 30 00:34:38.423542 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.423377 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f549cc40-a1ae-43aa-825f-11d538b9fa71-istiod-ca-cert" pod "f549cc40-a1ae-43aa-825f-11d538b9fa71" (UID: "f549cc40-a1ae-43aa-825f-11d538b9fa71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:34:38.424113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.424087 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-workload-certs" pod "f549cc40-a1ae-43aa-825f-11d538b9fa71" (UID: "f549cc40-a1ae-43aa-825f-11d538b9fa71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.424357 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.424102 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-credential-socket" pod "f549cc40-a1ae-43aa-825f-11d538b9fa71" (UID: "f549cc40-a1ae-43aa-825f-11d538b9fa71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.424357 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.424164 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-istio-data" pod "f549cc40-a1ae-43aa-825f-11d538b9fa71" (UID: "f549cc40-a1ae-43aa-825f-11d538b9fa71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.426128 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.426106 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.426265 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.426130 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.426265 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.426144 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.426265 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.426156 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.426265 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.426169 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.426265 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.426182 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f549cc40-a1ae-43aa-825f-11d538b9fa71-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.426265 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.426193 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.426265 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.426206 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tql2w\" (UniqueName: \"kubernetes.io/projected/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-kube-api-access-tql2w\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.426265 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.426220 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.426265 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.426233 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.426265 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.426247 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.426265 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.426258 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.426970 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.426273 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.426970 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.426286 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f93e25c-1a99-48c6-8a27-05e8ea723a6d-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.427243 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.427219 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f549cc40-a1ae-43aa-825f-11d538b9fa71-istio-podinfo" pod "f549cc40-a1ae-43aa-825f-11d538b9fa71" (UID: "f549cc40-a1ae-43aa-825f-11d538b9fa71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:34:38.427326 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.427282 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f549cc40-a1ae-43aa-825f-11d538b9fa71-kube-api-access-gx6ct" pod "f549cc40-a1ae-43aa-825f-11d538b9fa71" (UID: "f549cc40-a1ae-43aa-825f-11d538b9fa71"). InnerVolumeSpecName "kube-api-access-gx6ct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:38.427326 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.427307 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f549cc40-a1ae-43aa-825f-11d538b9fa71-istio-token" pod "f549cc40-a1ae-43aa-825f-11d538b9fa71" (UID: "f549cc40-a1ae-43aa-825f-11d538b9fa71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:38.427475 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.427229 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-istio-envoy" pod "f549cc40-a1ae-43aa-825f-11d538b9fa71" (UID: "f549cc40-a1ae-43aa-825f-11d538b9fa71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:38.534843 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.534799 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f549cc40-a1ae-43aa-825f-11d538b9fa71-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.535125 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.535100 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f549cc40-a1ae-43aa-825f-11d538b9fa71-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.535276 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.535262 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f549cc40-a1ae-43aa-825f-11d538b9fa71-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.535387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.535370 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx6ct\" (UniqueName: \"kubernetes.io/projected/f549cc40-a1ae-43aa-825f-11d538b9fa71-kube-api-access-gx6ct\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:38.743042 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.742962 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" Oct 30 00:34:38.743227 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.742960 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6" event={"ID":"b2cdf918-8248-4079-9823-830a84e927bf","Type":"ContainerDied","Data":"95e3e4f929c19d924f2f2ab534bb99e7d515caf9719740a8b252c265f6765ddd"} Oct 30 00:34:38.743227 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.743152 5001 scope.go:122] "RemoveContainer" containerID="69e51c74081b349c18d147927e847f86aedaf9f4774f2b7098fa342db180dc58" Oct 30 00:34:38.745869 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.745468 5001 generic.go:356] "Generic (PLEG): container finished" podID="9f93e25c-1a99-48c6-8a27-05e8ea723a6d" containerID="ce14b911b882bfb0ef52d120c36ffd835e196f7d259dc09b7991c7a79b099001" exitCode=0 Oct 30 00:34:38.745869 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.745554 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" event={"ID":"9f93e25c-1a99-48c6-8a27-05e8ea723a6d","Type":"ContainerDied","Data":"ce14b911b882bfb0ef52d120c36ffd835e196f7d259dc09b7991c7a79b099001"} Oct 30 00:34:38.745869 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.745561 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" Oct 30 00:34:38.745869 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.745597 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8" event={"ID":"9f93e25c-1a99-48c6-8a27-05e8ea723a6d","Type":"ContainerDied","Data":"b9f415b9ecbc2b5764e7d24cf4d50254ce5d78ae1de15c51dd41effee6602771"} Oct 30 00:34:38.750312 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.750289 5001 generic.go:356] "Generic (PLEG): container finished" podID="f549cc40-a1ae-43aa-825f-11d538b9fa71" containerID="d93b8a9f02863d14cd884a190ee8aebf784d7e11d8b488b2342554bccacab816" exitCode=0 Oct 30 00:34:38.750423 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.750345 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" Oct 30 00:34:38.750423 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.750374 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" event={"ID":"f549cc40-a1ae-43aa-825f-11d538b9fa71","Type":"ContainerDied","Data":"d93b8a9f02863d14cd884a190ee8aebf784d7e11d8b488b2342554bccacab816"} Oct 30 00:34:38.750550 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.750429 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s" event={"ID":"f549cc40-a1ae-43aa-825f-11d538b9fa71","Type":"ContainerDied","Data":"d3baae909e1014fd5313c7d5e01e1613d6aa5c8fefb782e73679e6c8457beee6"} Oct 30 00:34:38.763047 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.763026 5001 scope.go:122] "RemoveContainer" containerID="ce14b911b882bfb0ef52d120c36ffd835e196f7d259dc09b7991c7a79b099001" Oct 30 00:34:38.775113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.775073 5001 scope.go:122] "RemoveContainer" containerID="ce14b911b882bfb0ef52d120c36ffd835e196f7d259dc09b7991c7a79b099001" Oct 30 00:34:38.775402 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:34:38.775379 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce14b911b882bfb0ef52d120c36ffd835e196f7d259dc09b7991c7a79b099001\": container with ID starting with ce14b911b882bfb0ef52d120c36ffd835e196f7d259dc09b7991c7a79b099001 not found: ID does not exist" containerID="ce14b911b882bfb0ef52d120c36ffd835e196f7d259dc09b7991c7a79b099001" Oct 30 00:34:38.775504 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.775410 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce14b911b882bfb0ef52d120c36ffd835e196f7d259dc09b7991c7a79b099001"} err="failed to get container status \"ce14b911b882bfb0ef52d120c36ffd835e196f7d259dc09b7991c7a79b099001\": rpc error: code = NotFound desc = could not find container \"ce14b911b882bfb0ef52d120c36ffd835e196f7d259dc09b7991c7a79b099001\": container with ID starting with ce14b911b882bfb0ef52d120c36ffd835e196f7d259dc09b7991c7a79b099001 not found: ID does not exist" Oct 30 00:34:38.775504 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.775427 5001 scope.go:122] "RemoveContainer" containerID="d93b8a9f02863d14cd884a190ee8aebf784d7e11d8b488b2342554bccacab816" Oct 30 00:34:38.786593 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.786575 5001 scope.go:122] "RemoveContainer" containerID="d93b8a9f02863d14cd884a190ee8aebf784d7e11d8b488b2342554bccacab816" Oct 30 00:34:38.786866 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:34:38.786845 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d93b8a9f02863d14cd884a190ee8aebf784d7e11d8b488b2342554bccacab816\": container with ID starting with d93b8a9f02863d14cd884a190ee8aebf784d7e11d8b488b2342554bccacab816 not found: ID does not exist" containerID="d93b8a9f02863d14cd884a190ee8aebf784d7e11d8b488b2342554bccacab816" Oct 30 00:34:38.786924 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.786876 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d93b8a9f02863d14cd884a190ee8aebf784d7e11d8b488b2342554bccacab816"} err="failed to get container status \"d93b8a9f02863d14cd884a190ee8aebf784d7e11d8b488b2342554bccacab816\": rpc error: code = NotFound desc = could not find container \"d93b8a9f02863d14cd884a190ee8aebf784d7e11d8b488b2342554bccacab816\": container with ID starting with d93b8a9f02863d14cd884a190ee8aebf784d7e11d8b488b2342554bccacab816 not found: ID does not exist" Oct 30 00:34:38.804095 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.804070 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6"] Oct 30 00:34:38.813438 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.813408 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-xdhh6"] Oct 30 00:34:38.887642 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.887569 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s"] Oct 30 00:34:38.900531 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.899556 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-qqb5s"] Oct 30 00:34:38.957187 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.956744 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2cdf918-8248-4079-9823-830a84e927bf" path="/var/lib/kubelet/pods/b2cdf918-8248-4079-9823-830a84e927bf/volumes" Oct 30 00:34:38.957937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.957890 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f549cc40-a1ae-43aa-825f-11d538b9fa71" path="/var/lib/kubelet/pods/f549cc40-a1ae-43aa-825f-11d538b9fa71/volumes" Oct 30 00:34:38.958495 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.958470 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8"] Oct 30 00:34:38.958602 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:38.958525 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-564c8f-openshift-default-855fd5f755-rppr8"] Oct 30 00:34:40.029193 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:40.029159 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3eaadd9-xq2m8"] Oct 30 00:34:40.038605 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:40.038573 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3eaadd9-xq2m8"] Oct 30 00:34:40.920135 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:40.920099 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="109a1b04-b1d5-4d4e-ae59-c459fb040a6d" path="/var/lib/kubelet/pods/109a1b04-b1d5-4d4e-ae59-c459fb040a6d/volumes" Oct 30 00:34:40.920642 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:40.920627 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f93e25c-1a99-48c6-8a27-05e8ea723a6d" path="/var/lib/kubelet/pods/9f93e25c-1a99-48c6-8a27-05e8ea723a6d/volumes" Oct 30 00:34:41.483152 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:41.483112 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv" Oct 30 00:34:41.496223 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:41.496197 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv" Oct 30 00:34:41.798099 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:41.798063 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv" Oct 30 00:34:43.104033 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:43.103998 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5"] Oct 30 00:34:43.104549 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:43.104247 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" podUID="774a6da8-ca9f-4ca3-b132-9a449bb2d74c" containerName="istio-proxy" containerID="cri-o://53dc9b2a3e181c21c2532dd623eb0d739aabccee0de4de92297f2bead1638f3c" gracePeriod=30 Oct 30 00:34:43.135412 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:43.135378 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx"] Oct 30 00:34:43.135850 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:43.135820 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" podUID="aa2823a5-cf70-47df-8d48-430910c2023b" containerName="istio-proxy" containerID="cri-o://ea2e13577e89d03389a30c125dd7d5ff47cd19b80ea8c843324e3f5624104f5c" gracePeriod=30 Oct 30 00:34:43.143662 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:43.143633 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8"] Oct 30 00:34:43.144201 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:43.144172 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" podUID="9fddafb0-a134-4d5b-9956-0e9b925e799e" containerName="istio-proxy" containerID="cri-o://3a36f9090689231879333f6fcbfb765b00b58f238cb11efaae899dddbca2ded7" gracePeriod=30 Oct 30 00:34:45.083532 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:45.083499 5001 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:34:45.084085 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:45.083551 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" podUID="aa2823a5-cf70-47df-8d48-430910c2023b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:34:45.387515 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:45.387421 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8gb5_9fa688d8-29ca-4c1f-926a-f1a4099e6b44/console-operator/2.log" Oct 30 00:34:45.395214 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:45.395187 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8gb5_9fa688d8-29ca-4c1f-926a-f1a4099e6b44/console-operator/2.log" Oct 30 00:34:45.698413 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:45.698328 5001 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:34:45.698867 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:45.698837 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" podUID="774a6da8-ca9f-4ca3-b132-9a449bb2d74c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:34:47.030799 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:47.030759 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e4a714d-qwcz9"] Oct 30 00:34:47.039986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:47.039956 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-520095f9-xm2fs"] Oct 30 00:34:47.045778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:47.045752 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e4a714d-qwcz9"] Oct 30 00:34:47.050895 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:47.050872 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-520095f9-xm2fs"] Oct 30 00:34:47.343472 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:34:47.343380 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:34:48.167226 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.167190 5001 patch_prober.go:29] interesting pod/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 00:34:48.167645 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.167253 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" podUID="9fddafb0-a134-4d5b-9956-0e9b925e799e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 00:34:48.446990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.446954 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:34:48.523738 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.523710 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:34:48.527739 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.527715 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:34:48.535714 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.535364 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istio-podinfo\") pod \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " Oct 30 00:34:48.535714 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.535413 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istiod-ca-cert\") pod \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " Oct 30 00:34:48.535714 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.535528 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istio-data\") pod \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " Oct 30 00:34:48.535714 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.535570 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istio-token\" (UniqueName: \"kubernetes.io/projected/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istio-token\") pod \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " Oct 30 00:34:48.535714 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.535673 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istio-envoy\") pod \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " Oct 30 00:34:48.536182 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.535897 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-kube-api-access-z9q7m\" (UniqueName: \"kubernetes.io/projected/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-kube-api-access-z9q7m\") pod \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " Oct 30 00:34:48.536182 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.536057 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-workload-socket\") pod \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " Oct 30 00:34:48.536182 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.536148 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-credential-socket\") pod \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " Oct 30 00:34:48.536360 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.536279 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-workload-certs\") pod \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\" (UID: \"774a6da8-ca9f-4ca3-b132-9a449bb2d74c\") " Oct 30 00:34:48.537521 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.537369 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istiod-ca-cert" pod "774a6da8-ca9f-4ca3-b132-9a449bb2d74c" (UID: "774a6da8-ca9f-4ca3-b132-9a449bb2d74c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:34:48.537521 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.537497 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-workload-certs" pod "774a6da8-ca9f-4ca3-b132-9a449bb2d74c" (UID: "774a6da8-ca9f-4ca3-b132-9a449bb2d74c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.538866 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.538838 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-credential-socket" pod "774a6da8-ca9f-4ca3-b132-9a449bb2d74c" (UID: "774a6da8-ca9f-4ca3-b132-9a449bb2d74c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.539026 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.538935 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istio-podinfo" pod "774a6da8-ca9f-4ca3-b132-9a449bb2d74c" (UID: "774a6da8-ca9f-4ca3-b132-9a449bb2d74c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:34:48.539105 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.539073 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-workload-socket" pod "774a6da8-ca9f-4ca3-b132-9a449bb2d74c" (UID: "774a6da8-ca9f-4ca3-b132-9a449bb2d74c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.539711 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.539678 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istio-envoy" pod "774a6da8-ca9f-4ca3-b132-9a449bb2d74c" (UID: "774a6da8-ca9f-4ca3-b132-9a449bb2d74c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.539875 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.539839 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istio-data" pod "774a6da8-ca9f-4ca3-b132-9a449bb2d74c" (UID: "774a6da8-ca9f-4ca3-b132-9a449bb2d74c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.540189 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.540170 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istio-token" pod "774a6da8-ca9f-4ca3-b132-9a449bb2d74c" (UID: "774a6da8-ca9f-4ca3-b132-9a449bb2d74c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:48.541343 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.541327 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-kube-api-access-z9q7m" pod "774a6da8-ca9f-4ca3-b132-9a449bb2d74c" (UID: "774a6da8-ca9f-4ca3-b132-9a449bb2d74c"). InnerVolumeSpecName "kube-api-access-z9q7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:48.637770 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.637729 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa2823a5-cf70-47df-8d48-430910c2023b-istio-token\" (UniqueName: \"kubernetes.io/projected/aa2823a5-cf70-47df-8d48-430910c2023b-istio-token\") pod \"aa2823a5-cf70-47df-8d48-430910c2023b\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " Oct 30 00:34:48.637945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.637837 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-istio-data\") pod \"9fddafb0-a134-4d5b-9956-0e9b925e799e\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " Oct 30 00:34:48.637945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.637877 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-workload-certs\") pod \"aa2823a5-cf70-47df-8d48-430910c2023b\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " Oct 30 00:34:48.638054 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.638019 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa2823a5-cf70-47df-8d48-430910c2023b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2823a5-cf70-47df-8d48-430910c2023b-istio-podinfo\") pod \"aa2823a5-cf70-47df-8d48-430910c2023b\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " Oct 30 00:34:48.638101 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.638056 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-workload-certs\") pod \"9fddafb0-a134-4d5b-9956-0e9b925e799e\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " Oct 30 00:34:48.638144 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.638097 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-credential-socket\") pod \"9fddafb0-a134-4d5b-9956-0e9b925e799e\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " Oct 30 00:34:48.638189 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.638138 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-istio-envoy\") pod \"aa2823a5-cf70-47df-8d48-430910c2023b\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " Oct 30 00:34:48.638189 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.638171 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fddafb0-a134-4d5b-9956-0e9b925e799e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fddafb0-a134-4d5b-9956-0e9b925e799e-istio-podinfo\") pod \"9fddafb0-a134-4d5b-9956-0e9b925e799e\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " Oct 30 00:34:48.638264 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.638243 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-workload-socket\") pod \"aa2823a5-cf70-47df-8d48-430910c2023b\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " Oct 30 00:34:48.638342 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.638324 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fddafb0-a134-4d5b-9956-0e9b925e799e-istio-token\" (UniqueName: \"kubernetes.io/projected/9fddafb0-a134-4d5b-9956-0e9b925e799e-istio-token\") pod \"9fddafb0-a134-4d5b-9956-0e9b925e799e\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " Oct 30 00:34:48.638402 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.638386 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-credential-socket\") pod \"aa2823a5-cf70-47df-8d48-430910c2023b\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " Oct 30 00:34:48.638442 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.638424 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa2823a5-cf70-47df-8d48-430910c2023b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2823a5-cf70-47df-8d48-430910c2023b-istiod-ca-cert\") pod \"aa2823a5-cf70-47df-8d48-430910c2023b\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " Oct 30 00:34:48.638482 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.638451 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa2823a5-cf70-47df-8d48-430910c2023b-kube-api-access-42fpc\" (UniqueName: \"kubernetes.io/projected/aa2823a5-cf70-47df-8d48-430910c2023b-kube-api-access-42fpc\") pod \"aa2823a5-cf70-47df-8d48-430910c2023b\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " Oct 30 00:34:48.638531 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.638486 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-istio-envoy\") pod \"9fddafb0-a134-4d5b-9956-0e9b925e799e\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " Oct 30 00:34:48.638575 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.638562 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-istio-data\") pod \"aa2823a5-cf70-47df-8d48-430910c2023b\" (UID: \"aa2823a5-cf70-47df-8d48-430910c2023b\") " Oct 30 00:34:48.638681 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.638662 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fddafb0-a134-4d5b-9956-0e9b925e799e-kube-api-access-5krcv\" (UniqueName: \"kubernetes.io/projected/9fddafb0-a134-4d5b-9956-0e9b925e799e-kube-api-access-5krcv\") pod \"9fddafb0-a134-4d5b-9956-0e9b925e799e\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " Oct 30 00:34:48.638762 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.638727 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-workload-socket\") pod \"9fddafb0-a134-4d5b-9956-0e9b925e799e\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " Oct 30 00:34:48.638809 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.638777 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fddafb0-a134-4d5b-9956-0e9b925e799e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fddafb0-a134-4d5b-9956-0e9b925e799e-istiod-ca-cert\") pod \"9fddafb0-a134-4d5b-9956-0e9b925e799e\" (UID: \"9fddafb0-a134-4d5b-9956-0e9b925e799e\") " Oct 30 00:34:48.639602 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.639577 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.639698 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.639610 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.639698 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.639626 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.639698 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.639640 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.639698 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.639653 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.639698 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.639667 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.639698 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.639680 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9q7m\" (UniqueName: \"kubernetes.io/projected/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-kube-api-access-z9q7m\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.639698 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.639693 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.639978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.639707 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774a6da8-ca9f-4ca3-b132-9a449bb2d74c-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.640223 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.640195 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-workload-socket" pod "aa2823a5-cf70-47df-8d48-430910c2023b" (UID: "aa2823a5-cf70-47df-8d48-430910c2023b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.640415 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.640328 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-istio-data" pod "9fddafb0-a134-4d5b-9956-0e9b925e799e" (UID: "9fddafb0-a134-4d5b-9956-0e9b925e799e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.640787 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.640741 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-workload-certs" pod "9fddafb0-a134-4d5b-9956-0e9b925e799e" (UID: "9fddafb0-a134-4d5b-9956-0e9b925e799e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.640787 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.640745 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-workload-certs" pod "aa2823a5-cf70-47df-8d48-430910c2023b" (UID: "aa2823a5-cf70-47df-8d48-430910c2023b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.641015 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.640938 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fddafb0-a134-4d5b-9956-0e9b925e799e-istiod-ca-cert" pod "9fddafb0-a134-4d5b-9956-0e9b925e799e" (UID: "9fddafb0-a134-4d5b-9956-0e9b925e799e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:34:48.641099 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.641059 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-credential-socket" pod "aa2823a5-cf70-47df-8d48-430910c2023b" (UID: "aa2823a5-cf70-47df-8d48-430910c2023b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.641099 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.641086 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-credential-socket" pod "9fddafb0-a134-4d5b-9956-0e9b925e799e" (UID: "9fddafb0-a134-4d5b-9956-0e9b925e799e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.641237 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.641093 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa2823a5-cf70-47df-8d48-430910c2023b-istiod-ca-cert" pod "aa2823a5-cf70-47df-8d48-430910c2023b" (UID: "aa2823a5-cf70-47df-8d48-430910c2023b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:34:48.641595 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.641572 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-istio-data" pod "aa2823a5-cf70-47df-8d48-430910c2023b" (UID: "aa2823a5-cf70-47df-8d48-430910c2023b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.641706 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.641622 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-workload-socket" pod "9fddafb0-a134-4d5b-9956-0e9b925e799e" (UID: "9fddafb0-a134-4d5b-9956-0e9b925e799e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.644964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.644938 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa2823a5-cf70-47df-8d48-430910c2023b-istio-token" pod "aa2823a5-cf70-47df-8d48-430910c2023b" (UID: "aa2823a5-cf70-47df-8d48-430910c2023b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:48.645059 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.644984 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa2823a5-cf70-47df-8d48-430910c2023b-kube-api-access-42fpc" pod "aa2823a5-cf70-47df-8d48-430910c2023b" (UID: "aa2823a5-cf70-47df-8d48-430910c2023b"). InnerVolumeSpecName "kube-api-access-42fpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:48.645851 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.645824 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fddafb0-a134-4d5b-9956-0e9b925e799e-kube-api-access-5krcv" pod "9fddafb0-a134-4d5b-9956-0e9b925e799e" (UID: "9fddafb0-a134-4d5b-9956-0e9b925e799e"). InnerVolumeSpecName "kube-api-access-5krcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:48.647270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.647233 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fddafb0-a134-4d5b-9956-0e9b925e799e-istio-podinfo" pod "9fddafb0-a134-4d5b-9956-0e9b925e799e" (UID: "9fddafb0-a134-4d5b-9956-0e9b925e799e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:34:48.647270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.647248 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-istio-envoy" pod "aa2823a5-cf70-47df-8d48-430910c2023b" (UID: "aa2823a5-cf70-47df-8d48-430910c2023b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.647453 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.647310 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa2823a5-cf70-47df-8d48-430910c2023b-istio-podinfo" pod "aa2823a5-cf70-47df-8d48-430910c2023b" (UID: "aa2823a5-cf70-47df-8d48-430910c2023b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:34:48.647519 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.647451 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fddafb0-a134-4d5b-9956-0e9b925e799e-istio-token" pod "9fddafb0-a134-4d5b-9956-0e9b925e799e" (UID: "9fddafb0-a134-4d5b-9956-0e9b925e799e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:48.648576 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.648551 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-istio-envoy" pod "9fddafb0-a134-4d5b-9956-0e9b925e799e" (UID: "9fddafb0-a134-4d5b-9956-0e9b925e799e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:34:48.770545 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.770488 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.770545 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.770543 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fddafb0-a134-4d5b-9956-0e9b925e799e-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.770815 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.770559 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.770815 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.770574 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2823a5-cf70-47df-8d48-430910c2023b-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.770815 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.770588 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42fpc\" (UniqueName: \"kubernetes.io/projected/aa2823a5-cf70-47df-8d48-430910c2023b-kube-api-access-42fpc\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.770815 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.770604 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.770815 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.770619 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.770815 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.770631 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5krcv\" (UniqueName: \"kubernetes.io/projected/9fddafb0-a134-4d5b-9956-0e9b925e799e-kube-api-access-5krcv\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.770815 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.770646 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.770815 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.770659 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fddafb0-a134-4d5b-9956-0e9b925e799e-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.770815 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.770674 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa2823a5-cf70-47df-8d48-430910c2023b-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.770815 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.770686 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.770815 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.770698 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.770815 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.770711 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2823a5-cf70-47df-8d48-430910c2023b-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.770815 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.770722 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.770815 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.770737 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fddafb0-a134-4d5b-9956-0e9b925e799e-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.770815 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.770750 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2823a5-cf70-47df-8d48-430910c2023b-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.770815 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.770762 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fddafb0-a134-4d5b-9956-0e9b925e799e-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:48.865687 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.865651 5001 generic.go:356] "Generic (PLEG): container finished" podID="774a6da8-ca9f-4ca3-b132-9a449bb2d74c" containerID="53dc9b2a3e181c21c2532dd623eb0d739aabccee0de4de92297f2bead1638f3c" exitCode=0 Oct 30 00:34:48.865884 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.865702 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" Oct 30 00:34:48.865884 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.865732 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" event={"ID":"774a6da8-ca9f-4ca3-b132-9a449bb2d74c","Type":"ContainerDied","Data":"53dc9b2a3e181c21c2532dd623eb0d739aabccee0de4de92297f2bead1638f3c"} Oct 30 00:34:48.865884 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.865803 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5" event={"ID":"774a6da8-ca9f-4ca3-b132-9a449bb2d74c","Type":"ContainerDied","Data":"08b5c48fe9ca8a126166a66bdf7182880a01cb1d1c9f2f6c5a4f417f25e9579f"} Oct 30 00:34:48.865884 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.865833 5001 scope.go:122] "RemoveContainer" containerID="53dc9b2a3e181c21c2532dd623eb0d739aabccee0de4de92297f2bead1638f3c" Oct 30 00:34:48.867722 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.867700 5001 generic.go:356] "Generic (PLEG): container finished" podID="aa2823a5-cf70-47df-8d48-430910c2023b" containerID="ea2e13577e89d03389a30c125dd7d5ff47cd19b80ea8c843324e3f5624104f5c" exitCode=0 Oct 30 00:34:48.867849 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.867832 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" event={"ID":"aa2823a5-cf70-47df-8d48-430910c2023b","Type":"ContainerDied","Data":"ea2e13577e89d03389a30c125dd7d5ff47cd19b80ea8c843324e3f5624104f5c"} Oct 30 00:34:48.867972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.867877 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" event={"ID":"aa2823a5-cf70-47df-8d48-430910c2023b","Type":"ContainerDied","Data":"6e6309b8b073bcdb40d0b3772b49289acc1edef220c03687bcdb674950ed8f6f"} Oct 30 00:34:48.868055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.867974 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx" Oct 30 00:34:48.876478 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.876211 5001 generic.go:356] "Generic (PLEG): container finished" podID="9fddafb0-a134-4d5b-9956-0e9b925e799e" containerID="3a36f9090689231879333f6fcbfb765b00b58f238cb11efaae899dddbca2ded7" exitCode=0 Oct 30 00:34:48.876478 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.876386 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" event={"ID":"9fddafb0-a134-4d5b-9956-0e9b925e799e","Type":"ContainerDied","Data":"3a36f9090689231879333f6fcbfb765b00b58f238cb11efaae899dddbca2ded7"} Oct 30 00:34:48.876478 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.876434 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" event={"ID":"9fddafb0-a134-4d5b-9956-0e9b925e799e","Type":"ContainerDied","Data":"46e3154296fe1cca93e3b959f11a931319653db44d24d70ae2458ce8fdb6661a"} Oct 30 00:34:48.876706 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.876523 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8" Oct 30 00:34:48.886235 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.886215 5001 scope.go:122] "RemoveContainer" containerID="53dc9b2a3e181c21c2532dd623eb0d739aabccee0de4de92297f2bead1638f3c" Oct 30 00:34:48.886587 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:34:48.886559 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53dc9b2a3e181c21c2532dd623eb0d739aabccee0de4de92297f2bead1638f3c\": container with ID starting with 53dc9b2a3e181c21c2532dd623eb0d739aabccee0de4de92297f2bead1638f3c not found: ID does not exist" containerID="53dc9b2a3e181c21c2532dd623eb0d739aabccee0de4de92297f2bead1638f3c" Oct 30 00:34:48.886702 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.886595 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53dc9b2a3e181c21c2532dd623eb0d739aabccee0de4de92297f2bead1638f3c"} err="failed to get container status \"53dc9b2a3e181c21c2532dd623eb0d739aabccee0de4de92297f2bead1638f3c\": rpc error: code = NotFound desc = could not find container \"53dc9b2a3e181c21c2532dd623eb0d739aabccee0de4de92297f2bead1638f3c\": container with ID starting with 53dc9b2a3e181c21c2532dd623eb0d739aabccee0de4de92297f2bead1638f3c not found: ID does not exist" Oct 30 00:34:48.886702 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.886619 5001 scope.go:122] "RemoveContainer" containerID="ea2e13577e89d03389a30c125dd7d5ff47cd19b80ea8c843324e3f5624104f5c" Oct 30 00:34:48.902117 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.901984 5001 scope.go:122] "RemoveContainer" containerID="ea2e13577e89d03389a30c125dd7d5ff47cd19b80ea8c843324e3f5624104f5c" Oct 30 00:34:48.902407 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:34:48.902378 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea2e13577e89d03389a30c125dd7d5ff47cd19b80ea8c843324e3f5624104f5c\": container with ID starting with ea2e13577e89d03389a30c125dd7d5ff47cd19b80ea8c843324e3f5624104f5c not found: ID does not exist" containerID="ea2e13577e89d03389a30c125dd7d5ff47cd19b80ea8c843324e3f5624104f5c" Oct 30 00:34:48.902475 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.902420 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea2e13577e89d03389a30c125dd7d5ff47cd19b80ea8c843324e3f5624104f5c"} err="failed to get container status \"ea2e13577e89d03389a30c125dd7d5ff47cd19b80ea8c843324e3f5624104f5c\": rpc error: code = NotFound desc = could not find container \"ea2e13577e89d03389a30c125dd7d5ff47cd19b80ea8c843324e3f5624104f5c\": container with ID starting with ea2e13577e89d03389a30c125dd7d5ff47cd19b80ea8c843324e3f5624104f5c not found: ID does not exist" Oct 30 00:34:48.902475 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.902444 5001 scope.go:122] "RemoveContainer" containerID="3a36f9090689231879333f6fcbfb765b00b58f238cb11efaae899dddbca2ded7" Oct 30 00:34:48.920264 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.920243 5001 scope.go:122] "RemoveContainer" containerID="3a36f9090689231879333f6fcbfb765b00b58f238cb11efaae899dddbca2ded7" Oct 30 00:34:48.920651 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:34:48.920618 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a36f9090689231879333f6fcbfb765b00b58f238cb11efaae899dddbca2ded7\": container with ID starting with 3a36f9090689231879333f6fcbfb765b00b58f238cb11efaae899dddbca2ded7 not found: ID does not exist" containerID="3a36f9090689231879333f6fcbfb765b00b58f238cb11efaae899dddbca2ded7" Oct 30 00:34:48.920724 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.920656 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a36f9090689231879333f6fcbfb765b00b58f238cb11efaae899dddbca2ded7"} err="failed to get container status \"3a36f9090689231879333f6fcbfb765b00b58f238cb11efaae899dddbca2ded7\": rpc error: code = NotFound desc = could not find container \"3a36f9090689231879333f6fcbfb765b00b58f238cb11efaae899dddbca2ded7\": container with ID starting with 3a36f9090689231879333f6fcbfb765b00b58f238cb11efaae899dddbca2ded7 not found: ID does not exist" Oct 30 00:34:48.922965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.922939 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9559756a-57db-4a88-9a66-2889b5ead4af" path="/var/lib/kubelet/pods/9559756a-57db-4a88-9a66-2889b5ead4af/volumes" Oct 30 00:34:48.923496 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.923481 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f02d5c98-9a5f-4e40-b918-4c154e1e1c6b" path="/var/lib/kubelet/pods/f02d5c98-9a5f-4e40-b918-4c154e1e1c6b/volumes" Oct 30 00:34:48.940594 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.940566 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5"] Oct 30 00:34:48.947332 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.947308 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-kvck5"] Oct 30 00:34:48.973762 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.973730 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx"] Oct 30 00:34:48.982242 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:48.982212 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-qpdnx"] Oct 30 00:34:49.004890 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:49.004729 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8"] Oct 30 00:34:49.016461 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:49.016425 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1c718-openshift-default-76ddfb6779-7tth8"] Oct 30 00:34:50.920499 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:50.920467 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="774a6da8-ca9f-4ca3-b132-9a449bb2d74c" path="/var/lib/kubelet/pods/774a6da8-ca9f-4ca3-b132-9a449bb2d74c/volumes" Oct 30 00:34:50.920999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:50.920982 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fddafb0-a134-4d5b-9956-0e9b925e799e" path="/var/lib/kubelet/pods/9fddafb0-a134-4d5b-9956-0e9b925e799e/volumes" Oct 30 00:34:50.921442 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:50.921429 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa2823a5-cf70-47df-8d48-430910c2023b" path="/var/lib/kubelet/pods/aa2823a5-cf70-47df-8d48-430910c2023b/volumes" Oct 30 00:34:51.396546 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.396508 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv"] Oct 30 00:34:51.694883 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.684976 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be1f39-ffkkn"] Oct 30 00:34:51.694883 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.686144 5001 cpu_manager.go:414] "containerMap: removing container" podUID="9fddafb0-a134-4d5b-9956-0e9b925e799e" containerName="istio-proxy" Oct 30 00:34:51.694883 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.686167 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fddafb0-a134-4d5b-9956-0e9b925e799e" containerName="istio-proxy" Oct 30 00:34:51.694883 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.686188 5001 cpu_manager.go:414] "containerMap: removing container" podUID="9f93e25c-1a99-48c6-8a27-05e8ea723a6d" containerName="istio-proxy" Oct 30 00:34:51.694883 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.686196 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f93e25c-1a99-48c6-8a27-05e8ea723a6d" containerName="istio-proxy" Oct 30 00:34:51.694883 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.686215 5001 cpu_manager.go:414] "containerMap: removing container" podUID="774a6da8-ca9f-4ca3-b132-9a449bb2d74c" containerName="istio-proxy" Oct 30 00:34:51.694883 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.686223 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="774a6da8-ca9f-4ca3-b132-9a449bb2d74c" containerName="istio-proxy" Oct 30 00:34:51.694883 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.686244 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b2cdf918-8248-4079-9823-830a84e927bf" containerName="istio-proxy" Oct 30 00:34:51.694883 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.686251 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2cdf918-8248-4079-9823-830a84e927bf" containerName="istio-proxy" Oct 30 00:34:51.694883 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.686298 5001 cpu_manager.go:414] "containerMap: removing container" podUID="aa2823a5-cf70-47df-8d48-430910c2023b" containerName="istio-proxy" Oct 30 00:34:51.694883 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.686306 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa2823a5-cf70-47df-8d48-430910c2023b" containerName="istio-proxy" Oct 30 00:34:51.694883 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.686319 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f549cc40-a1ae-43aa-825f-11d538b9fa71" containerName="istio-proxy" Oct 30 00:34:51.694883 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.686328 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f549cc40-a1ae-43aa-825f-11d538b9fa71" containerName="istio-proxy" Oct 30 00:34:51.694883 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.686538 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa2823a5-cf70-47df-8d48-430910c2023b" containerName="istio-proxy" Oct 30 00:34:51.694883 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.686554 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f549cc40-a1ae-43aa-825f-11d538b9fa71" containerName="istio-proxy" Oct 30 00:34:51.694883 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.686572 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fddafb0-a134-4d5b-9956-0e9b925e799e" containerName="istio-proxy" Oct 30 00:34:51.694883 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.686582 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f93e25c-1a99-48c6-8a27-05e8ea723a6d" containerName="istio-proxy" Oct 30 00:34:51.694883 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.686590 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="774a6da8-ca9f-4ca3-b132-9a449bb2d74c" containerName="istio-proxy" Oct 30 00:34:51.694883 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.686602 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2cdf918-8248-4079-9823-830a84e927bf" containerName="istio-proxy" Oct 30 00:34:51.703303 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.703275 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be1f39-ffkkn" Oct 30 00:34:51.719028 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.713442 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be1f39-ffkkn"] Oct 30 00:34:51.719976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.719952 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:34:51.720434 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.720402 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:34:51.720564 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.720553 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:34:51.720657 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.720639 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:34:51.741032 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.740992 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67bf1dde-dae7-4990-9aeb-67b3aee9ef9c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-be1f39-ffkkn\" (UID: \"67bf1dde-dae7-4990-9aeb-67b3aee9ef9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be1f39-ffkkn" Oct 30 00:34:51.741316 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.741293 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67bf1dde-dae7-4990-9aeb-67b3aee9ef9c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-be1f39-ffkkn\" (UID: \"67bf1dde-dae7-4990-9aeb-67b3aee9ef9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be1f39-ffkkn" Oct 30 00:34:51.741409 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.741378 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl5sr\" (UniqueName: \"kubernetes.io/projected/67bf1dde-dae7-4990-9aeb-67b3aee9ef9c-kube-api-access-zl5sr\") pod \"maas-api-revoke-keys-e2e-explicit-be1f39-ffkkn\" (UID: \"67bf1dde-dae7-4990-9aeb-67b3aee9ef9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be1f39-ffkkn" Oct 30 00:34:51.842663 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.842627 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67bf1dde-dae7-4990-9aeb-67b3aee9ef9c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-be1f39-ffkkn\" (UID: \"67bf1dde-dae7-4990-9aeb-67b3aee9ef9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be1f39-ffkkn" Oct 30 00:34:51.842813 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.842672 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl5sr\" (UniqueName: \"kubernetes.io/projected/67bf1dde-dae7-4990-9aeb-67b3aee9ef9c-kube-api-access-zl5sr\") pod \"maas-api-revoke-keys-e2e-explicit-be1f39-ffkkn\" (UID: \"67bf1dde-dae7-4990-9aeb-67b3aee9ef9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be1f39-ffkkn" Oct 30 00:34:51.842887 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.842812 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67bf1dde-dae7-4990-9aeb-67b3aee9ef9c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-be1f39-ffkkn\" (UID: \"67bf1dde-dae7-4990-9aeb-67b3aee9ef9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be1f39-ffkkn" Oct 30 00:34:51.843475 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.843448 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67bf1dde-dae7-4990-9aeb-67b3aee9ef9c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-be1f39-ffkkn\" (UID: \"67bf1dde-dae7-4990-9aeb-67b3aee9ef9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be1f39-ffkkn" Oct 30 00:34:51.845085 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.845065 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67bf1dde-dae7-4990-9aeb-67b3aee9ef9c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-be1f39-ffkkn\" (UID: \"67bf1dde-dae7-4990-9aeb-67b3aee9ef9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be1f39-ffkkn" Oct 30 00:34:51.863669 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:51.863634 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl5sr\" (UniqueName: \"kubernetes.io/projected/67bf1dde-dae7-4990-9aeb-67b3aee9ef9c-kube-api-access-zl5sr\") pod \"maas-api-revoke-keys-e2e-explicit-be1f39-ffkkn\" (UID: \"67bf1dde-dae7-4990-9aeb-67b3aee9ef9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be1f39-ffkkn" Oct 30 00:34:52.033481 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:52.033449 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be1f39-ffkkn" Oct 30 00:34:52.442357 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:52.442327 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be1f39-ffkkn"] Oct 30 00:34:52.442617 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:34:52.442591 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67bf1dde_dae7_4990_9aeb_67b3aee9ef9c.slice/crio-8756a2917f0e3328c19e077c9115eeb17ce7526e4261c4d6ca911460342585fd WatchSource:0}: Error finding container 8756a2917f0e3328c19e077c9115eeb17ce7526e4261c4d6ca911460342585fd: Status 404 returned error can't find the container with id 8756a2917f0e3328c19e077c9115eeb17ce7526e4261c4d6ca911460342585fd Oct 30 00:34:52.936492 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:52.936453 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be1f39-ffkkn" event={"ID":"67bf1dde-dae7-4990-9aeb-67b3aee9ef9c","Type":"ContainerStarted","Data":"6b18862add77c626a7a42122b49ed976bb87da447170e9917c4dcc9df449ee7f"} Oct 30 00:34:52.936671 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:52.936529 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be1f39-ffkkn" event={"ID":"67bf1dde-dae7-4990-9aeb-67b3aee9ef9c","Type":"ContainerStarted","Data":"8756a2917f0e3328c19e077c9115eeb17ce7526e4261c4d6ca911460342585fd"} Oct 30 00:34:52.985940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:52.985795 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be1f39-ffkkn" podStartSLOduration=1.98577447 podStartE2EDuration="1.98577447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:52.975680471 +0000 UTC m=+1808.181181653" watchObservedRunningTime="2026-10-30 00:34:52.98577447 +0000 UTC m=+1808.191275651" Oct 30 00:34:53.958227 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:53.958195 5001 generic.go:356] "Generic (PLEG): container finished" podID="67bf1dde-dae7-4990-9aeb-67b3aee9ef9c" containerID="6b18862add77c626a7a42122b49ed976bb87da447170e9917c4dcc9df449ee7f" exitCode=0 Oct 30 00:34:53.958682 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:53.958292 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be1f39-ffkkn" event={"ID":"67bf1dde-dae7-4990-9aeb-67b3aee9ef9c","Type":"ContainerDied","Data":"6b18862add77c626a7a42122b49ed976bb87da447170e9917c4dcc9df449ee7f"} Oct 30 00:34:55.131220 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:55.131193 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be1f39-ffkkn" Oct 30 00:34:55.206132 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:55.206102 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67bf1dde-dae7-4990-9aeb-67b3aee9ef9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67bf1dde-dae7-4990-9aeb-67b3aee9ef9c-maas-api-openshift-ca\") pod \"67bf1dde-dae7-4990-9aeb-67b3aee9ef9c\" (UID: \"67bf1dde-dae7-4990-9aeb-67b3aee9ef9c\") " Oct 30 00:34:55.206337 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:55.206312 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67bf1dde-dae7-4990-9aeb-67b3aee9ef9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67bf1dde-dae7-4990-9aeb-67b3aee9ef9c-maas-api-xks-ca\") pod \"67bf1dde-dae7-4990-9aeb-67b3aee9ef9c\" (UID: \"67bf1dde-dae7-4990-9aeb-67b3aee9ef9c\") " Oct 30 00:34:55.206564 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:55.206540 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67bf1dde-dae7-4990-9aeb-67b3aee9ef9c-maas-api-openshift-ca" pod "67bf1dde-dae7-4990-9aeb-67b3aee9ef9c" (UID: "67bf1dde-dae7-4990-9aeb-67b3aee9ef9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:34:55.206717 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:55.206695 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67bf1dde-dae7-4990-9aeb-67b3aee9ef9c-kube-api-access-zl5sr\" (UniqueName: \"kubernetes.io/projected/67bf1dde-dae7-4990-9aeb-67b3aee9ef9c-kube-api-access-zl5sr\") pod \"67bf1dde-dae7-4990-9aeb-67b3aee9ef9c\" (UID: \"67bf1dde-dae7-4990-9aeb-67b3aee9ef9c\") " Oct 30 00:34:55.207556 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:55.207528 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67bf1dde-dae7-4990-9aeb-67b3aee9ef9c-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:55.208576 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:55.208548 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67bf1dde-dae7-4990-9aeb-67b3aee9ef9c-maas-api-xks-ca" pod "67bf1dde-dae7-4990-9aeb-67b3aee9ef9c" (UID: "67bf1dde-dae7-4990-9aeb-67b3aee9ef9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:34:55.208869 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:55.208838 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67bf1dde-dae7-4990-9aeb-67b3aee9ef9c-kube-api-access-zl5sr" pod "67bf1dde-dae7-4990-9aeb-67b3aee9ef9c" (UID: "67bf1dde-dae7-4990-9aeb-67b3aee9ef9c"). InnerVolumeSpecName "kube-api-access-zl5sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:34:55.308665 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:55.308632 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67bf1dde-dae7-4990-9aeb-67b3aee9ef9c-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:55.308665 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:55.308662 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl5sr\" (UniqueName: \"kubernetes.io/projected/67bf1dde-dae7-4990-9aeb-67b3aee9ef9c-kube-api-access-zl5sr\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:34:56.009589 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:56.009550 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be1f39-ffkkn" event={"ID":"67bf1dde-dae7-4990-9aeb-67b3aee9ef9c","Type":"ContainerDied","Data":"8756a2917f0e3328c19e077c9115eeb17ce7526e4261c4d6ca911460342585fd"} Oct 30 00:34:56.009869 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:56.009852 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8756a2917f0e3328c19e077c9115eeb17ce7526e4261c4d6ca911460342585fd" Oct 30 00:34:56.010063 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:56.010048 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be1f39-ffkkn" Oct 30 00:34:57.911421 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:57.911389 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl"] Oct 30 00:34:57.912539 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:57.912511 5001 cpu_manager.go:414] "containerMap: removing container" podUID="67bf1dde-dae7-4990-9aeb-67b3aee9ef9c" containerName="revoke-keys" Oct 30 00:34:57.912539 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:57.912535 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67bf1dde-dae7-4990-9aeb-67b3aee9ef9c" containerName="revoke-keys" Oct 30 00:34:57.912775 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:57.912760 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="67bf1dde-dae7-4990-9aeb-67b3aee9ef9c" containerName="revoke-keys" Oct 30 00:34:57.917056 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:57.917037 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl" Oct 30 00:34:57.937548 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:57.937516 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl"] Oct 30 00:34:57.940094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:57.940072 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-a782e5\"/\"kube-root-ca.crt\"" Oct 30 00:34:57.940230 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:57.940103 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-a782e5\"/\"openshift-service-ca.crt\"" Oct 30 00:34:57.940626 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:57.940603 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-a782e5\"/\"auth-test-model-a782e5-kserve-self-signed-certs\"" Oct 30 00:34:57.940967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:57.940906 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-a782e5\"/\"default-dockercfg-94mjr\"" Oct 30 00:34:57.969083 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:57.969052 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-dshm\") pod \"auth-test-model-a782e5-kserve-7596bffcbf-m68sl\" (UID: \"1be5133e-f463-4b8a-bc62-cd8a863ec1c7\") " pod="ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl" Oct 30 00:34:57.969242 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:57.969154 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-model-cache\") pod \"auth-test-model-a782e5-kserve-7596bffcbf-m68sl\" (UID: \"1be5133e-f463-4b8a-bc62-cd8a863ec1c7\") " pod="ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl" Oct 30 00:34:57.969242 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:57.969185 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnpg5\" (UniqueName: \"kubernetes.io/projected/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-kube-api-access-vnpg5\") pod \"auth-test-model-a782e5-kserve-7596bffcbf-m68sl\" (UID: \"1be5133e-f463-4b8a-bc62-cd8a863ec1c7\") " pod="ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl" Oct 30 00:34:57.969347 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:57.969267 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-tmp-dir\") pod \"auth-test-model-a782e5-kserve-7596bffcbf-m68sl\" (UID: \"1be5133e-f463-4b8a-bc62-cd8a863ec1c7\") " pod="ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl" Oct 30 00:34:57.969481 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:57.969462 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-home\") pod \"auth-test-model-a782e5-kserve-7596bffcbf-m68sl\" (UID: \"1be5133e-f463-4b8a-bc62-cd8a863ec1c7\") " pod="ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl" Oct 30 00:34:57.969625 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:57.969611 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-tls-certs\") pod \"auth-test-model-a782e5-kserve-7596bffcbf-m68sl\" (UID: \"1be5133e-f463-4b8a-bc62-cd8a863ec1c7\") " pod="ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl" Oct 30 00:34:58.037839 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:58.037811 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3536f0d4-tz9pw"] Oct 30 00:34:58.049178 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:58.049148 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7116507d-hx69n"] Oct 30 00:34:58.056343 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:58.056316 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3536f0d4-tz9pw"] Oct 30 00:34:58.061509 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:58.061483 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7116507d-hx69n"] Oct 30 00:34:58.071102 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:58.071076 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-tls-certs\") pod \"auth-test-model-a782e5-kserve-7596bffcbf-m68sl\" (UID: \"1be5133e-f463-4b8a-bc62-cd8a863ec1c7\") " pod="ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl" Oct 30 00:34:58.071215 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:58.071199 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-dshm\") pod \"auth-test-model-a782e5-kserve-7596bffcbf-m68sl\" (UID: \"1be5133e-f463-4b8a-bc62-cd8a863ec1c7\") " pod="ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl" Oct 30 00:34:58.071294 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:58.071278 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-model-cache\") pod \"auth-test-model-a782e5-kserve-7596bffcbf-m68sl\" (UID: \"1be5133e-f463-4b8a-bc62-cd8a863ec1c7\") " pod="ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl" Oct 30 00:34:58.071340 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:58.071309 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnpg5\" (UniqueName: \"kubernetes.io/projected/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-kube-api-access-vnpg5\") pod \"auth-test-model-a782e5-kserve-7596bffcbf-m68sl\" (UID: \"1be5133e-f463-4b8a-bc62-cd8a863ec1c7\") " pod="ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl" Oct 30 00:34:58.071415 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:58.071389 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-tmp-dir\") pod \"auth-test-model-a782e5-kserve-7596bffcbf-m68sl\" (UID: \"1be5133e-f463-4b8a-bc62-cd8a863ec1c7\") " pod="ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl" Oct 30 00:34:58.071562 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:58.071528 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-home\") pod \"auth-test-model-a782e5-kserve-7596bffcbf-m68sl\" (UID: \"1be5133e-f463-4b8a-bc62-cd8a863ec1c7\") " pod="ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl" Oct 30 00:34:58.071960 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:58.071827 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-tmp-dir\") pod \"auth-test-model-a782e5-kserve-7596bffcbf-m68sl\" (UID: \"1be5133e-f463-4b8a-bc62-cd8a863ec1c7\") " pod="ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl" Oct 30 00:34:58.071960 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:58.071897 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-model-cache\") pod \"auth-test-model-a782e5-kserve-7596bffcbf-m68sl\" (UID: \"1be5133e-f463-4b8a-bc62-cd8a863ec1c7\") " pod="ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl" Oct 30 00:34:58.072088 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:58.072003 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-home\") pod \"auth-test-model-a782e5-kserve-7596bffcbf-m68sl\" (UID: \"1be5133e-f463-4b8a-bc62-cd8a863ec1c7\") " pod="ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl" Oct 30 00:34:58.073995 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:58.073976 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-dshm\") pod \"auth-test-model-a782e5-kserve-7596bffcbf-m68sl\" (UID: \"1be5133e-f463-4b8a-bc62-cd8a863ec1c7\") " pod="ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl" Oct 30 00:34:58.074401 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:58.074380 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-tls-certs\") pod \"auth-test-model-a782e5-kserve-7596bffcbf-m68sl\" (UID: \"1be5133e-f463-4b8a-bc62-cd8a863ec1c7\") " pod="ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl" Oct 30 00:34:58.092207 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:58.092182 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnpg5\" (UniqueName: \"kubernetes.io/projected/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-kube-api-access-vnpg5\") pod \"auth-test-model-a782e5-kserve-7596bffcbf-m68sl\" (UID: \"1be5133e-f463-4b8a-bc62-cd8a863ec1c7\") " pod="ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl" Oct 30 00:34:58.248683 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:58.247277 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl" Oct 30 00:34:58.692247 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:34:58.692212 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1be5133e_f463_4b8a_bc62_cd8a863ec1c7.slice/crio-9be7bb56d9d9fc97866bf3a370ef5283e82415a18ea790ed7a68b2c267578a3e WatchSource:0}: Error finding container 9be7bb56d9d9fc97866bf3a370ef5283e82415a18ea790ed7a68b2c267578a3e: Status 404 returned error can't find the container with id 9be7bb56d9d9fc97866bf3a370ef5283e82415a18ea790ed7a68b2c267578a3e Oct 30 00:34:58.694077 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:58.694052 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl"] Oct 30 00:34:58.922572 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:58.922539 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838" path="/var/lib/kubelet/pods/4dc85ca4-ac2f-4fd2-96fe-ba547ae9c838/volumes" Oct 30 00:34:58.923094 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:58.923078 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b810194b-c717-4bf4-8558-07eab131bdc5" path="/var/lib/kubelet/pods/b810194b-c717-4bf4-8558-07eab131bdc5/volumes" Oct 30 00:34:59.073559 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:59.073520 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl" event={"ID":"1be5133e-f463-4b8a-bc62-cd8a863ec1c7","Type":"ContainerStarted","Data":"5ba3be446dee52c498e35d0691d39c9bbcc11a0410299efedaa4eb4e5fbb36bb"} Oct 30 00:34:59.073738 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:59.073568 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl" event={"ID":"1be5133e-f463-4b8a-bc62-cd8a863ec1c7","Type":"ContainerStarted","Data":"9be7bb56d9d9fc97866bf3a370ef5283e82415a18ea790ed7a68b2c267578a3e"} Oct 30 00:34:59.114399 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:59.114345 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl" podStartSLOduration=2.114331887 podStartE2EDuration="2.114331887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:59.108990188 +0000 UTC m=+1814.314491369" watchObservedRunningTime="2026-10-30 00:34:59.114331887 +0000 UTC m=+1814.319833079" Oct 30 00:34:59.253934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:34:59.250008 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl" Oct 30 00:35:00.711902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:00.711869 5001 scope.go:122] "RemoveContainer" containerID="1159211d6593cc026f268183bca70f915ab5e191fb5eaa1f0842fc5643dfd0cc" Oct 30 00:35:00.724087 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:00.724063 5001 scope.go:122] "RemoveContainer" containerID="afc79742de4339f9215ff9872320437f83deb06c02c494946c4aec766f5338f5" Oct 30 00:35:00.735428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:00.735406 5001 scope.go:122] "RemoveContainer" containerID="542329919c01f374ea9b12c20ad25bb165bb182f4727c28beba3cfce88b7fde0" Oct 30 00:35:00.745965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:00.745897 5001 scope.go:122] "RemoveContainer" containerID="426f05c55e3ce307838f2da9ff6e97ae53c026192023e0e7d1f26a6dfed3a6f1" Oct 30 00:35:00.756729 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:00.756709 5001 scope.go:122] "RemoveContainer" containerID="c3f694f3e3083c848b84982973eb54a7c2f6b48458a0dbbca9b0302250e74c69" Oct 30 00:35:00.766945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:00.766899 5001 scope.go:122] "RemoveContainer" containerID="8bd58dae50647ae9a0e5cf08f38a2c23dea86049ad08365a933028d61635c2b7" Oct 30 00:35:00.777739 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:00.777714 5001 scope.go:122] "RemoveContainer" containerID="9c9744aba98cca178108371d9aa83cc4c1f16c71db773c7e239c9254e952ed35" Oct 30 00:35:00.788115 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:00.788093 5001 scope.go:122] "RemoveContainer" containerID="9dd4a7bf9238ef86a2f0459715c4836910a54cfa4f1e60b4395e9fb739250bdd" Oct 30 00:35:06.439001 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:06.430293 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv" podUID="b2d9f394-c972-4b00-ba20-6bd04d977827" containerName="main" containerID="cri-o://f4b3c20a953dea467b583c9e56500c2b10276b746805331b09820c9009ccbd14" gracePeriod=45 Oct 30 00:35:07.072986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:07.072257 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv" Oct 30 00:35:07.107511 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:07.106472 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2d9f394-c972-4b00-ba20-6bd04d977827-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f394-c972-4b00-ba20-6bd04d977827-tmp-dir\") pod \"b2d9f394-c972-4b00-ba20-6bd04d977827\" (UID: \"b2d9f394-c972-4b00-ba20-6bd04d977827\") " Oct 30 00:35:07.107511 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:07.106605 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2d9f394-c972-4b00-ba20-6bd04d977827-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f394-c972-4b00-ba20-6bd04d977827-model-cache\") pod \"b2d9f394-c972-4b00-ba20-6bd04d977827\" (UID: \"b2d9f394-c972-4b00-ba20-6bd04d977827\") " Oct 30 00:35:07.107511 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:07.106668 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2d9f394-c972-4b00-ba20-6bd04d977827-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f394-c972-4b00-ba20-6bd04d977827-dshm\") pod \"b2d9f394-c972-4b00-ba20-6bd04d977827\" (UID: \"b2d9f394-c972-4b00-ba20-6bd04d977827\") " Oct 30 00:35:07.107511 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:07.106694 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2d9f394-c972-4b00-ba20-6bd04d977827-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2d9f394-c972-4b00-ba20-6bd04d977827-tls-certs\") pod \"b2d9f394-c972-4b00-ba20-6bd04d977827\" (UID: \"b2d9f394-c972-4b00-ba20-6bd04d977827\") " Oct 30 00:35:07.107511 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:07.106781 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2d9f394-c972-4b00-ba20-6bd04d977827-home\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f394-c972-4b00-ba20-6bd04d977827-home\") pod \"b2d9f394-c972-4b00-ba20-6bd04d977827\" (UID: \"b2d9f394-c972-4b00-ba20-6bd04d977827\") " Oct 30 00:35:07.107511 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:07.106819 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2d9f394-c972-4b00-ba20-6bd04d977827-kube-api-access-l97gb\" (UniqueName: \"kubernetes.io/projected/b2d9f394-c972-4b00-ba20-6bd04d977827-kube-api-access-l97gb\") pod \"b2d9f394-c972-4b00-ba20-6bd04d977827\" (UID: \"b2d9f394-c972-4b00-ba20-6bd04d977827\") " Oct 30 00:35:07.107511 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:07.106858 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d9f394-c972-4b00-ba20-6bd04d977827-model-cache" pod "b2d9f394-c972-4b00-ba20-6bd04d977827" (UID: "b2d9f394-c972-4b00-ba20-6bd04d977827"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:35:07.107511 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:07.106926 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d9f394-c972-4b00-ba20-6bd04d977827-tmp-dir" pod "b2d9f394-c972-4b00-ba20-6bd04d977827" (UID: "b2d9f394-c972-4b00-ba20-6bd04d977827"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:35:07.107511 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:07.107187 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d9f394-c972-4b00-ba20-6bd04d977827-home" pod "b2d9f394-c972-4b00-ba20-6bd04d977827" (UID: "b2d9f394-c972-4b00-ba20-6bd04d977827"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:35:07.107511 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:07.107442 5001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f394-c972-4b00-ba20-6bd04d977827-model-cache\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:35:07.107511 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:07.107462 5001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f394-c972-4b00-ba20-6bd04d977827-home\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:35:07.107511 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:07.107471 5001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f394-c972-4b00-ba20-6bd04d977827-tmp-dir\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:35:07.144687 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:07.144619 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2d9f394-c972-4b00-ba20-6bd04d977827-tls-certs" pod "b2d9f394-c972-4b00-ba20-6bd04d977827" (UID: "b2d9f394-c972-4b00-ba20-6bd04d977827"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:35:07.151938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:07.150079 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2d9f394-c972-4b00-ba20-6bd04d977827-kube-api-access-l97gb" pod "b2d9f394-c972-4b00-ba20-6bd04d977827" (UID: "b2d9f394-c972-4b00-ba20-6bd04d977827"). InnerVolumeSpecName "kube-api-access-l97gb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:35:07.181374 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:07.181081 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d9f394-c972-4b00-ba20-6bd04d977827-dshm" pod "b2d9f394-c972-4b00-ba20-6bd04d977827" (UID: "b2d9f394-c972-4b00-ba20-6bd04d977827"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:35:07.209104 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:07.209014 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l97gb\" (UniqueName: \"kubernetes.io/projected/b2d9f394-c972-4b00-ba20-6bd04d977827-kube-api-access-l97gb\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:35:07.209104 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:07.209051 5001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f394-c972-4b00-ba20-6bd04d977827-dshm\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:35:07.209104 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:07.209066 5001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2d9f394-c972-4b00-ba20-6bd04d977827-tls-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:35:07.278985 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:07.278946 5001 generic.go:356] "Generic (PLEG): container finished" podID="b2d9f394-c972-4b00-ba20-6bd04d977827" containerID="f4b3c20a953dea467b583c9e56500c2b10276b746805331b09820c9009ccbd14" exitCode=0 Oct 30 00:35:07.279686 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:07.279215 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv" event={"ID":"b2d9f394-c972-4b00-ba20-6bd04d977827","Type":"ContainerDied","Data":"f4b3c20a953dea467b583c9e56500c2b10276b746805331b09820c9009ccbd14"} Oct 30 00:35:07.279931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:07.279897 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv" event={"ID":"b2d9f394-c972-4b00-ba20-6bd04d977827","Type":"ContainerDied","Data":"7623b40e98b1d1dd0ecadaa3bba1094342af29bbfa3ec7e4e1bd83e68e113451"} Oct 30 00:35:07.280065 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:07.280040 5001 scope.go:122] "RemoveContainer" containerID="f4b3c20a953dea467b583c9e56500c2b10276b746805331b09820c9009ccbd14" Oct 30 00:35:07.280287 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:07.279287 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv" Oct 30 00:35:07.292848 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:07.292753 5001 scope.go:122] "RemoveContainer" containerID="f4b3c20a953dea467b583c9e56500c2b10276b746805331b09820c9009ccbd14" Oct 30 00:35:07.293507 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:35:07.293480 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b3c20a953dea467b583c9e56500c2b10276b746805331b09820c9009ccbd14\": container with ID starting with f4b3c20a953dea467b583c9e56500c2b10276b746805331b09820c9009ccbd14 not found: ID does not exist" containerID="f4b3c20a953dea467b583c9e56500c2b10276b746805331b09820c9009ccbd14" Oct 30 00:35:07.293598 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:07.293518 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b3c20a953dea467b583c9e56500c2b10276b746805331b09820c9009ccbd14"} err="failed to get container status \"f4b3c20a953dea467b583c9e56500c2b10276b746805331b09820c9009ccbd14\": rpc error: code = NotFound desc = could not find container \"f4b3c20a953dea467b583c9e56500c2b10276b746805331b09820c9009ccbd14\": container with ID starting with f4b3c20a953dea467b583c9e56500c2b10276b746805331b09820c9009ccbd14 not found: ID does not exist" Oct 30 00:35:07.343991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:07.342981 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv"] Oct 30 00:35:07.362935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:07.357871 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv"] Oct 30 00:35:07.787859 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:07.787820 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-explicit-be1f39/explicit-model-be1f39-kserve-6bc4555567-8mkkv" podUID="b2d9f394-c972-4b00-ba20-6bd04d977827" containerName="main" probeResult="failure" output="Get \"https://10.132.1.27:8000/ready\": context deadline exceeded" Oct 30 00:35:08.261139 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:08.260029 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl" Oct 30 00:35:08.278141 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:08.278115 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl" Oct 30 00:35:08.303825 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:08.303800 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl" Oct 30 00:35:08.963306 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:08.963268 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2d9f394-c972-4b00-ba20-6bd04d977827" path="/var/lib/kubelet/pods/b2d9f394-c972-4b00-ba20-6bd04d977827/volumes" Oct 30 00:35:11.069950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:11.069871 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe1626e5-fh6kd"] Oct 30 00:35:11.075861 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:11.075830 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe1626e5-fh6kd"] Oct 30 00:35:12.921296 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:12.921259 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16378f33-f8e3-4810-b03a-e4e67067d5a3" path="/var/lib/kubelet/pods/16378f33-f8e3-4810-b03a-e4e67067d5a3/volumes" Oct 30 00:35:15.404996 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:15.404903 5001 generic.go:356] "Generic (PLEG): container finished" podID="4deafe88-91e1-4878-9203-e28de81ce261" containerID="933325317e94c64b0c6c337585297e652db8cab8364924dd9ede8a26aa107a3d" exitCode=0 Oct 30 00:35:15.404996 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:15.404975 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" event={"ID":"4deafe88-91e1-4878-9203-e28de81ce261","Type":"ContainerDied","Data":"933325317e94c64b0c6c337585297e652db8cab8364924dd9ede8a26aa107a3d"} Oct 30 00:35:15.405471 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:15.405455 5001 scope.go:122] "RemoveContainer" containerID="933325317e94c64b0c6c337585297e652db8cab8364924dd9ede8a26aa107a3d" Oct 30 00:35:15.407053 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:15.407034 5001 generic.go:356] "Generic (PLEG): container finished" podID="ff8dd458-3a31-4084-b71e-2ecb4114c4e0" containerID="90b44cb7797ab9211767e050a7f01e06de111cf75c35871faa72b7c2e1f8587e" exitCode=0 Oct 30 00:35:15.407158 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:15.407097 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" event={"ID":"ff8dd458-3a31-4084-b71e-2ecb4114c4e0","Type":"ContainerDied","Data":"90b44cb7797ab9211767e050a7f01e06de111cf75c35871faa72b7c2e1f8587e"} Oct 30 00:35:15.407275 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:15.407244 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:35:15.407343 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:15.407327 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:35:15.407384 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:15.407367 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:35:15.407466 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:15.407452 5001 scope.go:122] "RemoveContainer" containerID="90b44cb7797ab9211767e050a7f01e06de111cf75c35871faa72b7c2e1f8587e" Oct 30 00:35:15.409028 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:15.409000 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:35:15.409117 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:15.409078 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:35:15.409117 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:15.409110 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:35:16.446931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:16.446607 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" event={"ID":"4deafe88-91e1-4878-9203-e28de81ce261","Type":"ContainerStarted","Data":"977bc8320a7d587e835c0ecebace3f3e8744acd49deb8a2609f106465549c600"} Oct 30 00:35:16.458933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:16.450997 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" event={"ID":"ff8dd458-3a31-4084-b71e-2ecb4114c4e0","Type":"ContainerStarted","Data":"8d892ea84ae30a92515fdd50f2e9fa4e8b07d234db4e759d796600fb8423bc0e"} Oct 30 00:35:16.491195 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:16.491164 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:35:16.491513 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:16.491487 5001 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Oct 30 00:35:16.491628 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:16.491552 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" podUID="ff8dd458-3a31-4084-b71e-2ecb4114c4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Oct 30 00:35:16.654935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:16.652074 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:35:16.655263 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:16.655234 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 30 00:35:16.655440 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:16.655416 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" podUID="4deafe88-91e1-4878-9203-e28de81ce261" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 30 00:35:16.716995 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:16.713627 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:35:17.373000 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:35:17.372961 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:35:17.492478 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:17.492339 5001 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Oct 30 00:35:17.492478 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:17.492385 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" podUID="ff8dd458-3a31-4084-b71e-2ecb4114c4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Oct 30 00:35:17.654371 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:17.654272 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 30 00:35:17.654669 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:17.654635 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" podUID="4deafe88-91e1-4878-9203-e28de81ce261" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 30 00:35:18.492245 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:18.492206 5001 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Oct 30 00:35:18.492540 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:18.492514 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" podUID="ff8dd458-3a31-4084-b71e-2ecb4114c4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Oct 30 00:35:18.651360 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:18.651324 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 30 00:35:18.651629 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:18.651604 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" podUID="4deafe88-91e1-4878-9203-e28de81ce261" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 30 00:35:19.493099 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:19.493058 5001 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Oct 30 00:35:19.494024 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:19.493995 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" podUID="ff8dd458-3a31-4084-b71e-2ecb4114c4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Oct 30 00:35:19.658400 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:19.658367 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 30 00:35:19.658643 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:19.658618 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" podUID="4deafe88-91e1-4878-9203-e28de81ce261" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 30 00:35:20.492245 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:20.492216 5001 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Oct 30 00:35:20.492469 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:20.492449 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" podUID="ff8dd458-3a31-4084-b71e-2ecb4114c4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Oct 30 00:35:20.656947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:20.656460 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 30 00:35:20.656947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:20.656517 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" podUID="4deafe88-91e1-4878-9203-e28de81ce261" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 30 00:35:21.492239 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:21.492209 5001 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Oct 30 00:35:21.492506 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:21.492481 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" podUID="ff8dd458-3a31-4084-b71e-2ecb4114c4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Oct 30 00:35:21.606514 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:21.606480 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx"] Oct 30 00:35:21.607544 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:21.607516 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b2d9f394-c972-4b00-ba20-6bd04d977827" containerName="main" Oct 30 00:35:21.607748 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:21.607731 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2d9f394-c972-4b00-ba20-6bd04d977827" containerName="main" Oct 30 00:35:21.608171 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:21.608148 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2d9f394-c972-4b00-ba20-6bd04d977827" containerName="main" Oct 30 00:35:21.658318 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:21.658277 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 30 00:35:21.658822 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:21.658352 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" podUID="4deafe88-91e1-4878-9203-e28de81ce261" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 30 00:35:21.659175 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:21.659146 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx" Oct 30 00:35:21.663778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:21.663634 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx"] Oct 30 00:35:21.672792 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:21.672770 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-63e288\"/\"auth-test-model-63e288-kserve-self-signed-certs\"" Oct 30 00:35:21.675097 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:21.675067 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-63e288\"/\"openshift-service-ca.crt\"" Oct 30 00:35:21.677315 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:21.677297 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-63e288\"/\"default-dockercfg-t49ct\"" Oct 30 00:35:21.677718 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:21.677700 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-63e288\"/\"kube-root-ca.crt\"" Oct 30 00:35:21.779580 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:21.779535 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/218cceb3-ff29-4eba-bd54-d8ea3885aae2-tls-certs\") pod \"auth-test-model-63e288-kserve-79fc495b49-7s6rx\" (UID: \"218cceb3-ff29-4eba-bd54-d8ea3885aae2\") " pod="ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx" Oct 30 00:35:21.779764 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:21.779651 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xvqp\" (UniqueName: \"kubernetes.io/projected/218cceb3-ff29-4eba-bd54-d8ea3885aae2-kube-api-access-2xvqp\") pod \"auth-test-model-63e288-kserve-79fc495b49-7s6rx\" (UID: \"218cceb3-ff29-4eba-bd54-d8ea3885aae2\") " pod="ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx" Oct 30 00:35:21.779764 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:21.779750 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/218cceb3-ff29-4eba-bd54-d8ea3885aae2-tmp-dir\") pod \"auth-test-model-63e288-kserve-79fc495b49-7s6rx\" (UID: \"218cceb3-ff29-4eba-bd54-d8ea3885aae2\") " pod="ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx" Oct 30 00:35:21.779848 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:21.779779 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/218cceb3-ff29-4eba-bd54-d8ea3885aae2-home\") pod \"auth-test-model-63e288-kserve-79fc495b49-7s6rx\" (UID: \"218cceb3-ff29-4eba-bd54-d8ea3885aae2\") " pod="ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx" Oct 30 00:35:21.779848 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:21.779805 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/218cceb3-ff29-4eba-bd54-d8ea3885aae2-model-cache\") pod \"auth-test-model-63e288-kserve-79fc495b49-7s6rx\" (UID: \"218cceb3-ff29-4eba-bd54-d8ea3885aae2\") " pod="ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx" Oct 30 00:35:21.779937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:21.779881 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/218cceb3-ff29-4eba-bd54-d8ea3885aae2-dshm\") pod \"auth-test-model-63e288-kserve-79fc495b49-7s6rx\" (UID: \"218cceb3-ff29-4eba-bd54-d8ea3885aae2\") " pod="ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx" Oct 30 00:35:21.887659 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:21.887609 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/218cceb3-ff29-4eba-bd54-d8ea3885aae2-dshm\") pod \"auth-test-model-63e288-kserve-79fc495b49-7s6rx\" (UID: \"218cceb3-ff29-4eba-bd54-d8ea3885aae2\") " pod="ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx" Oct 30 00:35:21.897570 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:21.897524 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/218cceb3-ff29-4eba-bd54-d8ea3885aae2-tls-certs\") pod \"auth-test-model-63e288-kserve-79fc495b49-7s6rx\" (UID: \"218cceb3-ff29-4eba-bd54-d8ea3885aae2\") " pod="ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx" Oct 30 00:35:21.898506 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:21.898478 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xvqp\" (UniqueName: \"kubernetes.io/projected/218cceb3-ff29-4eba-bd54-d8ea3885aae2-kube-api-access-2xvqp\") pod \"auth-test-model-63e288-kserve-79fc495b49-7s6rx\" (UID: \"218cceb3-ff29-4eba-bd54-d8ea3885aae2\") " pod="ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx" Oct 30 00:35:21.907123 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:21.907090 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/218cceb3-ff29-4eba-bd54-d8ea3885aae2-tmp-dir\") pod \"auth-test-model-63e288-kserve-79fc495b49-7s6rx\" (UID: \"218cceb3-ff29-4eba-bd54-d8ea3885aae2\") " pod="ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx" Oct 30 00:35:21.907745 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:21.907724 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/218cceb3-ff29-4eba-bd54-d8ea3885aae2-home\") pod \"auth-test-model-63e288-kserve-79fc495b49-7s6rx\" (UID: \"218cceb3-ff29-4eba-bd54-d8ea3885aae2\") " pod="ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx" Oct 30 00:35:21.908044 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:21.908022 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/218cceb3-ff29-4eba-bd54-d8ea3885aae2-model-cache\") pod \"auth-test-model-63e288-kserve-79fc495b49-7s6rx\" (UID: \"218cceb3-ff29-4eba-bd54-d8ea3885aae2\") " pod="ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx" Oct 30 00:35:21.925935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:21.908623 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/218cceb3-ff29-4eba-bd54-d8ea3885aae2-model-cache\") pod \"auth-test-model-63e288-kserve-79fc495b49-7s6rx\" (UID: \"218cceb3-ff29-4eba-bd54-d8ea3885aae2\") " pod="ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx" Oct 30 00:35:21.926835 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:21.910503 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/218cceb3-ff29-4eba-bd54-d8ea3885aae2-dshm\") pod \"auth-test-model-63e288-kserve-79fc495b49-7s6rx\" (UID: \"218cceb3-ff29-4eba-bd54-d8ea3885aae2\") " pod="ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx" Oct 30 00:35:21.927039 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:21.907676 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/218cceb3-ff29-4eba-bd54-d8ea3885aae2-tmp-dir\") pod \"auth-test-model-63e288-kserve-79fc495b49-7s6rx\" (UID: \"218cceb3-ff29-4eba-bd54-d8ea3885aae2\") " pod="ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx" Oct 30 00:35:21.927151 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:21.910810 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/218cceb3-ff29-4eba-bd54-d8ea3885aae2-home\") pod \"auth-test-model-63e288-kserve-79fc495b49-7s6rx\" (UID: \"218cceb3-ff29-4eba-bd54-d8ea3885aae2\") " pod="ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx" Oct 30 00:35:21.949941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:21.940129 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/218cceb3-ff29-4eba-bd54-d8ea3885aae2-tls-certs\") pod \"auth-test-model-63e288-kserve-79fc495b49-7s6rx\" (UID: \"218cceb3-ff29-4eba-bd54-d8ea3885aae2\") " pod="ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx" Oct 30 00:35:21.951030 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:21.951003 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xvqp\" (UniqueName: \"kubernetes.io/projected/218cceb3-ff29-4eba-bd54-d8ea3885aae2-kube-api-access-2xvqp\") pod \"auth-test-model-63e288-kserve-79fc495b49-7s6rx\" (UID: \"218cceb3-ff29-4eba-bd54-d8ea3885aae2\") " pod="ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx" Oct 30 00:35:21.983250 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:21.983204 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx" Oct 30 00:35:22.469879 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:35:22.469847 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod218cceb3_ff29_4eba_bd54_d8ea3885aae2.slice/crio-cbf1de0687f80fa55409c4234be1fff81adacea8d4512186ffff8fa55d795d92 WatchSource:0}: Error finding container cbf1de0687f80fa55409c4234be1fff81adacea8d4512186ffff8fa55d795d92: Status 404 returned error can't find the container with id cbf1de0687f80fa55409c4234be1fff81adacea8d4512186ffff8fa55d795d92 Oct 30 00:35:22.470047 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:22.469966 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx"] Oct 30 00:35:22.499100 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:22.499071 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:35:22.499435 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:22.499379 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:35:22.500267 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:22.500247 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:35:22.568494 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:22.568462 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx" event={"ID":"218cceb3-ff29-4eba-bd54-d8ea3885aae2","Type":"ContainerStarted","Data":"cbf1de0687f80fa55409c4234be1fff81adacea8d4512186ffff8fa55d795d92"} Oct 30 00:35:22.659241 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:22.659209 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:35:22.660842 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:22.660816 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:35:23.554942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:23.550207 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be1f39-ffkkn"] Oct 30 00:35:23.586938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:23.586061 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be1f39-ffkkn"] Oct 30 00:35:23.646935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:23.646164 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx" event={"ID":"218cceb3-ff29-4eba-bd54-d8ea3885aae2","Type":"ContainerStarted","Data":"45c8be50728258ee937214ebd2da0186f9ad3adff412b2ae18e6202e4e6b4e38"} Oct 30 00:35:23.693783 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:23.693270 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx" podStartSLOduration=2.693250941 podStartE2EDuration="2.693250941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:35:23.691495609 +0000 UTC m=+1838.896996792" watchObservedRunningTime="2026-10-30 00:35:23.693250941 +0000 UTC m=+1838.898752124" Oct 30 00:35:23.984373 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:23.984291 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx" Oct 30 00:35:24.922563 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:24.922529 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67bf1dde-dae7-4990-9aeb-67b3aee9ef9c" path="/var/lib/kubelet/pods/67bf1dde-dae7-4990-9aeb-67b3aee9ef9c/volumes" Oct 30 00:35:31.388738 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:31.388535 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb"] Oct 30 00:35:31.389207 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:31.388809 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" podUID="85331496-4439-48d1-aca8-19dc5cf91bae" containerName="istio-proxy" containerID="cri-o://472a381bccf53ec6b6af885f3d3685480bf630ac169e07b4eb8cc3696c5b6d91" gracePeriod=30 Oct 30 00:35:31.984619 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:31.984586 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx" Oct 30 00:35:31.997424 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:31.997401 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx" Oct 30 00:35:32.029881 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:32.029848 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26375c9f-nh57j"] Oct 30 00:35:32.035005 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:32.034981 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26375c9f-nh57j"] Oct 30 00:35:32.893985 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:32.889642 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx" Oct 30 00:35:32.955521 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:32.955486 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bee48652-da0e-4181-93b1-5e88cc6511d4" path="/var/lib/kubelet/pods/bee48652-da0e-4181-93b1-5e88cc6511d4/volumes" Oct 30 00:35:35.207617 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:35.207576 5001 patch_prober.go:29] interesting pod/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:35:35.208063 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:35.207637 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" podUID="85331496-4439-48d1-aca8-19dc5cf91bae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:35:36.679780 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.679755 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:35:36.752678 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.752641 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85331496-4439-48d1-aca8-19dc5cf91bae-kube-api-access-lblr5\" (UniqueName: \"kubernetes.io/projected/85331496-4439-48d1-aca8-19dc5cf91bae-kube-api-access-lblr5\") pod \"85331496-4439-48d1-aca8-19dc5cf91bae\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " Oct 30 00:35:36.752880 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.752713 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-credential-socket\") pod \"85331496-4439-48d1-aca8-19dc5cf91bae\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " Oct 30 00:35:36.752880 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.752786 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85331496-4439-48d1-aca8-19dc5cf91bae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85331496-4439-48d1-aca8-19dc5cf91bae-istiod-ca-cert\") pod \"85331496-4439-48d1-aca8-19dc5cf91bae\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " Oct 30 00:35:36.752880 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.752817 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-istio-data\") pod \"85331496-4439-48d1-aca8-19dc5cf91bae\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " Oct 30 00:35:36.752880 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.752868 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85331496-4439-48d1-aca8-19dc5cf91bae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85331496-4439-48d1-aca8-19dc5cf91bae-istio-podinfo\") pod \"85331496-4439-48d1-aca8-19dc5cf91bae\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " Oct 30 00:35:36.753166 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.753023 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85331496-4439-48d1-aca8-19dc5cf91bae-istio-token\" (UniqueName: \"kubernetes.io/projected/85331496-4439-48d1-aca8-19dc5cf91bae-istio-token\") pod \"85331496-4439-48d1-aca8-19dc5cf91bae\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " Oct 30 00:35:36.753166 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.753090 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-istio-envoy\") pod \"85331496-4439-48d1-aca8-19dc5cf91bae\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " Oct 30 00:35:36.753277 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.753167 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-workload-certs\") pod \"85331496-4439-48d1-aca8-19dc5cf91bae\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " Oct 30 00:35:36.753277 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.753199 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-workload-socket\") pod \"85331496-4439-48d1-aca8-19dc5cf91bae\" (UID: \"85331496-4439-48d1-aca8-19dc5cf91bae\") " Oct 30 00:35:36.753662 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.753405 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-credential-socket" pod "85331496-4439-48d1-aca8-19dc5cf91bae" (UID: "85331496-4439-48d1-aca8-19dc5cf91bae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:35:36.753998 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.753860 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-workload-certs" pod "85331496-4439-48d1-aca8-19dc5cf91bae" (UID: "85331496-4439-48d1-aca8-19dc5cf91bae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:35:36.754098 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.754071 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-istio-data" pod "85331496-4439-48d1-aca8-19dc5cf91bae" (UID: "85331496-4439-48d1-aca8-19dc5cf91bae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:35:36.754254 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.754135 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:35:36.754254 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.754158 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:35:36.754404 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.754249 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85331496-4439-48d1-aca8-19dc5cf91bae-istiod-ca-cert" pod "85331496-4439-48d1-aca8-19dc5cf91bae" (UID: "85331496-4439-48d1-aca8-19dc5cf91bae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:35:36.754609 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.754575 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-workload-socket" pod "85331496-4439-48d1-aca8-19dc5cf91bae" (UID: "85331496-4439-48d1-aca8-19dc5cf91bae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:35:36.755878 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.755854 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85331496-4439-48d1-aca8-19dc5cf91bae-kube-api-access-lblr5" pod "85331496-4439-48d1-aca8-19dc5cf91bae" (UID: "85331496-4439-48d1-aca8-19dc5cf91bae"). InnerVolumeSpecName "kube-api-access-lblr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:35:36.756013 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.755893 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85331496-4439-48d1-aca8-19dc5cf91bae-istio-token" pod "85331496-4439-48d1-aca8-19dc5cf91bae" (UID: "85331496-4439-48d1-aca8-19dc5cf91bae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:35:36.756478 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.756448 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85331496-4439-48d1-aca8-19dc5cf91bae-istio-podinfo" pod "85331496-4439-48d1-aca8-19dc5cf91bae" (UID: "85331496-4439-48d1-aca8-19dc5cf91bae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:35:36.757068 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.757041 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-istio-envoy" pod "85331496-4439-48d1-aca8-19dc5cf91bae" (UID: "85331496-4439-48d1-aca8-19dc5cf91bae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:35:36.855786 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.855752 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lblr5\" (UniqueName: \"kubernetes.io/projected/85331496-4439-48d1-aca8-19dc5cf91bae-kube-api-access-lblr5\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:35:36.855786 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.855781 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85331496-4439-48d1-aca8-19dc5cf91bae-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:35:36.855786 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.855792 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:35:36.856029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.855806 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85331496-4439-48d1-aca8-19dc5cf91bae-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:35:36.856029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.855815 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85331496-4439-48d1-aca8-19dc5cf91bae-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:35:36.856029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.855824 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:35:36.856029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.855833 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85331496-4439-48d1-aca8-19dc5cf91bae-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:35:36.915341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.915242 5001 generic.go:356] "Generic (PLEG): container finished" podID="85331496-4439-48d1-aca8-19dc5cf91bae" containerID="472a381bccf53ec6b6af885f3d3685480bf630ac169e07b4eb8cc3696c5b6d91" exitCode=0 Oct 30 00:35:36.915527 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.915358 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" Oct 30 00:35:36.920006 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.919980 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" event={"ID":"85331496-4439-48d1-aca8-19dc5cf91bae","Type":"ContainerDied","Data":"472a381bccf53ec6b6af885f3d3685480bf630ac169e07b4eb8cc3696c5b6d91"} Oct 30 00:35:36.920149 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.920022 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb" event={"ID":"85331496-4439-48d1-aca8-19dc5cf91bae","Type":"ContainerDied","Data":"74cc6485ce61a9589f3ca4d2c7a2243edc993dbe698c157e47714ea1416ed702"} Oct 30 00:35:36.920149 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.920045 5001 scope.go:122] "RemoveContainer" containerID="472a381bccf53ec6b6af885f3d3685480bf630ac169e07b4eb8cc3696c5b6d91" Oct 30 00:35:36.933921 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.933892 5001 scope.go:122] "RemoveContainer" containerID="3cbe2ffbe68f15f32a979205b3d35ab61a6e5e80e49806a3543017ac7ec0895e" Oct 30 00:35:36.948279 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.948258 5001 scope.go:122] "RemoveContainer" containerID="472a381bccf53ec6b6af885f3d3685480bf630ac169e07b4eb8cc3696c5b6d91" Oct 30 00:35:36.948627 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:35:36.948603 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"472a381bccf53ec6b6af885f3d3685480bf630ac169e07b4eb8cc3696c5b6d91\": container with ID starting with 472a381bccf53ec6b6af885f3d3685480bf630ac169e07b4eb8cc3696c5b6d91 not found: ID does not exist" containerID="472a381bccf53ec6b6af885f3d3685480bf630ac169e07b4eb8cc3696c5b6d91" Oct 30 00:35:36.948731 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.948633 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"472a381bccf53ec6b6af885f3d3685480bf630ac169e07b4eb8cc3696c5b6d91"} err="failed to get container status \"472a381bccf53ec6b6af885f3d3685480bf630ac169e07b4eb8cc3696c5b6d91\": rpc error: code = NotFound desc = could not find container \"472a381bccf53ec6b6af885f3d3685480bf630ac169e07b4eb8cc3696c5b6d91\": container with ID starting with 472a381bccf53ec6b6af885f3d3685480bf630ac169e07b4eb8cc3696c5b6d91 not found: ID does not exist" Oct 30 00:35:36.948731 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.948649 5001 scope.go:122] "RemoveContainer" containerID="3cbe2ffbe68f15f32a979205b3d35ab61a6e5e80e49806a3543017ac7ec0895e" Oct 30 00:35:36.949022 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:35:36.948990 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cbe2ffbe68f15f32a979205b3d35ab61a6e5e80e49806a3543017ac7ec0895e\": container with ID starting with 3cbe2ffbe68f15f32a979205b3d35ab61a6e5e80e49806a3543017ac7ec0895e not found: ID does not exist" containerID="3cbe2ffbe68f15f32a979205b3d35ab61a6e5e80e49806a3543017ac7ec0895e" Oct 30 00:35:36.949153 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.949027 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cbe2ffbe68f15f32a979205b3d35ab61a6e5e80e49806a3543017ac7ec0895e"} err="failed to get container status \"3cbe2ffbe68f15f32a979205b3d35ab61a6e5e80e49806a3543017ac7ec0895e\": rpc error: code = NotFound desc = could not find container \"3cbe2ffbe68f15f32a979205b3d35ab61a6e5e80e49806a3543017ac7ec0895e\": container with ID starting with 3cbe2ffbe68f15f32a979205b3d35ab61a6e5e80e49806a3543017ac7ec0895e not found: ID does not exist" Oct 30 00:35:36.965362 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.965336 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb"] Oct 30 00:35:36.970451 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:36.970428 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-be1f39-openshift-default-65d5fccd9c-hxjdb"] Oct 30 00:35:38.920005 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:38.919969 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85331496-4439-48d1-aca8-19dc5cf91bae" path="/var/lib/kubelet/pods/85331496-4439-48d1-aca8-19dc5cf91bae/volumes" Oct 30 00:35:46.031901 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:46.031861 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4895cea6-5nmqx"] Oct 30 00:35:46.041331 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:46.041295 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4895cea6-5nmqx"] Oct 30 00:35:46.920214 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:46.920181 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01383a75-9ce0-4813-ae3b-2ac0fb292132" path="/var/lib/kubelet/pods/01383a75-9ce0-4813-ae3b-2ac0fb292132/volumes" Oct 30 00:35:47.348843 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:35:47.348808 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:35:54.116334 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:54.116298 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz"] Oct 30 00:35:54.117601 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:54.117573 5001 cpu_manager.go:414] "containerMap: removing container" podUID="85331496-4439-48d1-aca8-19dc5cf91bae" containerName="istio-proxy" Oct 30 00:35:54.117601 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:54.117600 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85331496-4439-48d1-aca8-19dc5cf91bae" containerName="istio-proxy" Oct 30 00:35:54.117803 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:54.117625 5001 cpu_manager.go:414] "containerMap: removing container" podUID="85331496-4439-48d1-aca8-19dc5cf91bae" containerName="istio-proxy" Oct 30 00:35:54.117803 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:54.117632 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85331496-4439-48d1-aca8-19dc5cf91bae" containerName="istio-proxy" Oct 30 00:35:54.117960 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:54.117867 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="85331496-4439-48d1-aca8-19dc5cf91bae" containerName="istio-proxy" Oct 30 00:35:54.117960 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:54.117884 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="85331496-4439-48d1-aca8-19dc5cf91bae" containerName="istio-proxy" Oct 30 00:35:54.122985 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:54.121718 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz" Oct 30 00:35:54.143832 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:54.143805 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-05c6b2\"/\"default-dockercfg-mpq4p\"" Oct 30 00:35:54.146181 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:54.146084 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-05c6b2\"/\"kube-root-ca.crt\"" Oct 30 00:35:54.146181 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:54.146143 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-05c6b2\"/\"auth-test-model-05c6b2-kserve-self-signed-certs\"" Oct 30 00:35:54.146505 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:54.146477 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-05c6b2\"/\"openshift-service-ca.crt\"" Oct 30 00:35:54.153807 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:54.153782 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz"] Oct 30 00:35:54.156561 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:54.156535 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2326ccd2-e034-479e-9ffd-6366f1327831-tmp-dir\") pod \"auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz\" (UID: \"2326ccd2-e034-479e-9ffd-6366f1327831\") " pod="ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz" Oct 30 00:35:54.156679 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:54.156570 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2326ccd2-e034-479e-9ffd-6366f1327831-home\") pod \"auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz\" (UID: \"2326ccd2-e034-479e-9ffd-6366f1327831\") " pod="ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz" Oct 30 00:35:54.156679 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:54.156670 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2326ccd2-e034-479e-9ffd-6366f1327831-model-cache\") pod \"auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz\" (UID: \"2326ccd2-e034-479e-9ffd-6366f1327831\") " pod="ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz" Oct 30 00:35:54.156822 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:54.156794 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2326ccd2-e034-479e-9ffd-6366f1327831-tls-certs\") pod \"auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz\" (UID: \"2326ccd2-e034-479e-9ffd-6366f1327831\") " pod="ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz" Oct 30 00:35:54.156889 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:54.156863 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-456zj\" (UniqueName: \"kubernetes.io/projected/2326ccd2-e034-479e-9ffd-6366f1327831-kube-api-access-456zj\") pod \"auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz\" (UID: \"2326ccd2-e034-479e-9ffd-6366f1327831\") " pod="ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz" Oct 30 00:35:54.157015 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:54.156988 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2326ccd2-e034-479e-9ffd-6366f1327831-dshm\") pod \"auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz\" (UID: \"2326ccd2-e034-479e-9ffd-6366f1327831\") " pod="ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz" Oct 30 00:35:54.258045 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:54.258010 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2326ccd2-e034-479e-9ffd-6366f1327831-tls-certs\") pod \"auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz\" (UID: \"2326ccd2-e034-479e-9ffd-6366f1327831\") " pod="ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz" Oct 30 00:35:54.258244 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:54.258092 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-456zj\" (UniqueName: \"kubernetes.io/projected/2326ccd2-e034-479e-9ffd-6366f1327831-kube-api-access-456zj\") pod \"auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz\" (UID: \"2326ccd2-e034-479e-9ffd-6366f1327831\") " pod="ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz" Oct 30 00:35:54.258244 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:54.258156 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2326ccd2-e034-479e-9ffd-6366f1327831-dshm\") pod \"auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz\" (UID: \"2326ccd2-e034-479e-9ffd-6366f1327831\") " pod="ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz" Oct 30 00:35:54.258244 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:54.258219 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2326ccd2-e034-479e-9ffd-6366f1327831-tmp-dir\") pod \"auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz\" (UID: \"2326ccd2-e034-479e-9ffd-6366f1327831\") " pod="ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz" Oct 30 00:35:54.258434 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:54.258247 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2326ccd2-e034-479e-9ffd-6366f1327831-home\") pod \"auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz\" (UID: \"2326ccd2-e034-479e-9ffd-6366f1327831\") " pod="ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz" Oct 30 00:35:54.258434 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:54.258310 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2326ccd2-e034-479e-9ffd-6366f1327831-model-cache\") pod \"auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz\" (UID: \"2326ccd2-e034-479e-9ffd-6366f1327831\") " pod="ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz" Oct 30 00:35:54.258738 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:54.258685 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2326ccd2-e034-479e-9ffd-6366f1327831-tmp-dir\") pod \"auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz\" (UID: \"2326ccd2-e034-479e-9ffd-6366f1327831\") " pod="ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz" Oct 30 00:35:54.258738 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:54.258721 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2326ccd2-e034-479e-9ffd-6366f1327831-home\") pod \"auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz\" (UID: \"2326ccd2-e034-479e-9ffd-6366f1327831\") " pod="ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz" Oct 30 00:35:54.259008 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:54.258764 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2326ccd2-e034-479e-9ffd-6366f1327831-model-cache\") pod \"auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz\" (UID: \"2326ccd2-e034-479e-9ffd-6366f1327831\") " pod="ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz" Oct 30 00:35:54.260645 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:54.260624 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2326ccd2-e034-479e-9ffd-6366f1327831-dshm\") pod \"auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz\" (UID: \"2326ccd2-e034-479e-9ffd-6366f1327831\") " pod="ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz" Oct 30 00:35:54.260883 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:54.260860 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2326ccd2-e034-479e-9ffd-6366f1327831-tls-certs\") pod \"auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz\" (UID: \"2326ccd2-e034-479e-9ffd-6366f1327831\") " pod="ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz" Oct 30 00:35:54.279654 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:54.279619 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-456zj\" (UniqueName: \"kubernetes.io/projected/2326ccd2-e034-479e-9ffd-6366f1327831-kube-api-access-456zj\") pod \"auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz\" (UID: \"2326ccd2-e034-479e-9ffd-6366f1327831\") " pod="ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz" Oct 30 00:35:54.440401 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:54.438214 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz" Oct 30 00:35:54.891320 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:54.890996 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz"] Oct 30 00:35:55.124598 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:55.124554 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz" event={"ID":"2326ccd2-e034-479e-9ffd-6366f1327831","Type":"ContainerStarted","Data":"bff93ef65e145b762c31958f3fa5ecab1e14ec53f6dc81aada2019e5f7ca2a77"} Oct 30 00:35:55.124598 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:55.124604 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz" event={"ID":"2326ccd2-e034-479e-9ffd-6366f1327831","Type":"ContainerStarted","Data":"fc15c4cbc0a31a5a5842de879fdb6807095cf988d68c906314e4742f9952a620"} Oct 30 00:35:55.164567 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:55.164465 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz" podStartSLOduration=1.164451305 podStartE2EDuration="1.164451305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:35:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:35:55.160052152 +0000 UTC m=+1870.365553333" watchObservedRunningTime="2026-10-30 00:35:55.164451305 +0000 UTC m=+1870.369952485" Oct 30 00:35:55.439496 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:55.439410 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz" Oct 30 00:35:56.378625 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:56.378590 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn"] Oct 30 00:35:56.382395 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:56.382372 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn" Oct 30 00:35:56.406219 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:56.404853 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-415d24\"/\"default-dockercfg-s86l2\"" Oct 30 00:35:56.406219 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:56.404858 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-415d24\"/\"kube-root-ca.crt\"" Oct 30 00:35:56.406219 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:56.404884 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-415d24\"/\"auth-test-model-415d24-kserve-self-signed-certs\"" Oct 30 00:35:56.406219 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:56.405019 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-415d24\"/\"openshift-service-ca.crt\"" Oct 30 00:35:56.420201 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:56.420170 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn"] Oct 30 00:35:56.568252 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:56.568213 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdd8c42e-35c2-4141-afab-e400b23a9163-model-cache\") pod \"auth-test-model-415d24-kserve-cd4d594f7-d6ksn\" (UID: \"cdd8c42e-35c2-4141-afab-e400b23a9163\") " pod="ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn" Oct 30 00:35:56.568438 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:56.568349 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdd8c42e-35c2-4141-afab-e400b23a9163-dshm\") pod \"auth-test-model-415d24-kserve-cd4d594f7-d6ksn\" (UID: \"cdd8c42e-35c2-4141-afab-e400b23a9163\") " pod="ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn" Oct 30 00:35:56.568438 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:56.568389 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdd8c42e-35c2-4141-afab-e400b23a9163-tmp-dir\") pod \"auth-test-model-415d24-kserve-cd4d594f7-d6ksn\" (UID: \"cdd8c42e-35c2-4141-afab-e400b23a9163\") " pod="ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn" Oct 30 00:35:56.568540 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:56.568484 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdd8c42e-35c2-4141-afab-e400b23a9163-tls-certs\") pod \"auth-test-model-415d24-kserve-cd4d594f7-d6ksn\" (UID: \"cdd8c42e-35c2-4141-afab-e400b23a9163\") " pod="ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn" Oct 30 00:35:56.568595 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:56.568582 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdd8c42e-35c2-4141-afab-e400b23a9163-home\") pod \"auth-test-model-415d24-kserve-cd4d594f7-d6ksn\" (UID: \"cdd8c42e-35c2-4141-afab-e400b23a9163\") " pod="ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn" Oct 30 00:35:56.568658 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:56.568644 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4mf2\" (UniqueName: \"kubernetes.io/projected/cdd8c42e-35c2-4141-afab-e400b23a9163-kube-api-access-f4mf2\") pod \"auth-test-model-415d24-kserve-cd4d594f7-d6ksn\" (UID: \"cdd8c42e-35c2-4141-afab-e400b23a9163\") " pod="ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn" Oct 30 00:35:56.672289 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:56.672198 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdd8c42e-35c2-4141-afab-e400b23a9163-tls-certs\") pod \"auth-test-model-415d24-kserve-cd4d594f7-d6ksn\" (UID: \"cdd8c42e-35c2-4141-afab-e400b23a9163\") " pod="ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn" Oct 30 00:35:56.673270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:56.673228 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdd8c42e-35c2-4141-afab-e400b23a9163-home\") pod \"auth-test-model-415d24-kserve-cd4d594f7-d6ksn\" (UID: \"cdd8c42e-35c2-4141-afab-e400b23a9163\") " pod="ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn" Oct 30 00:35:56.673434 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:56.673320 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4mf2\" (UniqueName: \"kubernetes.io/projected/cdd8c42e-35c2-4141-afab-e400b23a9163-kube-api-access-f4mf2\") pod \"auth-test-model-415d24-kserve-cd4d594f7-d6ksn\" (UID: \"cdd8c42e-35c2-4141-afab-e400b23a9163\") " pod="ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn" Oct 30 00:35:56.673434 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:56.673400 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdd8c42e-35c2-4141-afab-e400b23a9163-model-cache\") pod \"auth-test-model-415d24-kserve-cd4d594f7-d6ksn\" (UID: \"cdd8c42e-35c2-4141-afab-e400b23a9163\") " pod="ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn" Oct 30 00:35:56.673569 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:56.673504 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdd8c42e-35c2-4141-afab-e400b23a9163-dshm\") pod \"auth-test-model-415d24-kserve-cd4d594f7-d6ksn\" (UID: \"cdd8c42e-35c2-4141-afab-e400b23a9163\") " pod="ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn" Oct 30 00:35:56.673569 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:56.673552 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdd8c42e-35c2-4141-afab-e400b23a9163-tmp-dir\") pod \"auth-test-model-415d24-kserve-cd4d594f7-d6ksn\" (UID: \"cdd8c42e-35c2-4141-afab-e400b23a9163\") " pod="ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn" Oct 30 00:35:56.686931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:56.674047 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdd8c42e-35c2-4141-afab-e400b23a9163-tmp-dir\") pod \"auth-test-model-415d24-kserve-cd4d594f7-d6ksn\" (UID: \"cdd8c42e-35c2-4141-afab-e400b23a9163\") " pod="ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn" Oct 30 00:35:56.687338 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:56.684900 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdd8c42e-35c2-4141-afab-e400b23a9163-home\") pod \"auth-test-model-415d24-kserve-cd4d594f7-d6ksn\" (UID: \"cdd8c42e-35c2-4141-afab-e400b23a9163\") " pod="ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn" Oct 30 00:35:56.687489 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:56.685155 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdd8c42e-35c2-4141-afab-e400b23a9163-model-cache\") pod \"auth-test-model-415d24-kserve-cd4d594f7-d6ksn\" (UID: \"cdd8c42e-35c2-4141-afab-e400b23a9163\") " pod="ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn" Oct 30 00:35:56.696988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:56.696860 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdd8c42e-35c2-4141-afab-e400b23a9163-dshm\") pod \"auth-test-model-415d24-kserve-cd4d594f7-d6ksn\" (UID: \"cdd8c42e-35c2-4141-afab-e400b23a9163\") " pod="ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn" Oct 30 00:35:56.719754 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:56.719669 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4mf2\" (UniqueName: \"kubernetes.io/projected/cdd8c42e-35c2-4141-afab-e400b23a9163-kube-api-access-f4mf2\") pod \"auth-test-model-415d24-kserve-cd4d594f7-d6ksn\" (UID: \"cdd8c42e-35c2-4141-afab-e400b23a9163\") " pod="ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn" Oct 30 00:35:56.720689 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:56.720653 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdd8c42e-35c2-4141-afab-e400b23a9163-tls-certs\") pod \"auth-test-model-415d24-kserve-cd4d594f7-d6ksn\" (UID: \"cdd8c42e-35c2-4141-afab-e400b23a9163\") " pod="ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn" Oct 30 00:35:57.000463 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:57.000365 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn" Oct 30 00:35:57.312397 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:35:57.312367 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcdd8c42e_35c2_4141_afab_e400b23a9163.slice/crio-c4da90fddd593ef94872f459347146658a09b1151b403a49dd80b0ec5b14ecee WatchSource:0}: Error finding container c4da90fddd593ef94872f459347146658a09b1151b403a49dd80b0ec5b14ecee: Status 404 returned error can't find the container with id c4da90fddd593ef94872f459347146658a09b1151b403a49dd80b0ec5b14ecee Oct 30 00:35:57.314449 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:57.314428 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn"] Oct 30 00:35:58.156424 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:58.156382 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn" event={"ID":"cdd8c42e-35c2-4141-afab-e400b23a9163","Type":"ContainerStarted","Data":"63e5202d8c75d8622d61709ccbcd213eb2a8777a9567cf02b520968278bb99c2"} Oct 30 00:35:58.156424 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:58.156428 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn" event={"ID":"cdd8c42e-35c2-4141-afab-e400b23a9163","Type":"ContainerStarted","Data":"c4da90fddd593ef94872f459347146658a09b1151b403a49dd80b0ec5b14ecee"} Oct 30 00:35:58.197390 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:58.197339 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn" podStartSLOduration=2.197326643 podStartE2EDuration="2.197326643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:35:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:35:58.192995376 +0000 UTC m=+1873.398496558" watchObservedRunningTime="2026-10-30 00:35:58.197326643 +0000 UTC m=+1873.402827856" Oct 30 00:35:59.000940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:35:59.000885 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn" Oct 30 00:36:00.936933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:00.924666 5001 scope.go:122] "RemoveContainer" containerID="fde3a2a46256e507ee8b8a26271c9a8a2a61e5a9854224981f5a622f00a82105" Oct 30 00:36:00.976895 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:00.976872 5001 scope.go:122] "RemoveContainer" containerID="639dfaddabb608fb8c564e1f006985ba8e03981b71ec847a9f5ea60f8f64161f" Oct 30 00:36:00.987676 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:00.987654 5001 scope.go:122] "RemoveContainer" containerID="53d9777a34b3bc557e08469f2f9aa5b0280b67fd22676f5867d1219462aa3f6c" Oct 30 00:36:00.998333 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:00.998311 5001 scope.go:122] "RemoveContainer" containerID="840ccaea6e91f6ee8a6e7a21dc1e73379fa16c47e6a6392e15e41759be12012f" Oct 30 00:36:01.008441 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:01.008421 5001 scope.go:122] "RemoveContainer" containerID="df59fad72286f3a4f28ab5fd93f26d9b390383b8e4ebb66ffd7da0fe40a551bc" Oct 30 00:36:01.018485 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:01.018461 5001 scope.go:122] "RemoveContainer" containerID="ba63a7f2ece412f209dd5010fa6ae6e60e02f6370967b3dd8850665cc69ca53d" Oct 30 00:36:01.027964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:01.027946 5001 scope.go:122] "RemoveContainer" containerID="586b8ff2639d4721d508befbfa08b0229036a480297c281591653a54547104b0" Oct 30 00:36:01.037843 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:01.037824 5001 scope.go:122] "RemoveContainer" containerID="fcbcc2caa4003d7edff0d1e3e44f8669d2603c74ddfb980f60d12efe776142b5" Oct 30 00:36:04.439544 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:04.439506 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz" Oct 30 00:36:04.451815 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:04.451792 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz" Oct 30 00:36:05.258311 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:05.258279 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz" Oct 30 00:36:07.001283 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:07.001252 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn" Oct 30 00:36:07.016733 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:07.016705 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn" Oct 30 00:36:07.280074 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:07.280044 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn" Oct 30 00:36:17.327970 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:36:17.327890 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:36:25.204417 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:25.204380 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ae68a59-h7r2h"] Oct 30 00:36:25.227020 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:25.226986 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ae68a59-h7r2h"] Oct 30 00:36:26.920319 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:26.920287 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a43cdd7-b9e2-4c02-b4c8-bf785b583213" path="/var/lib/kubelet/pods/4a43cdd7-b9e2-4c02-b4c8-bf785b583213/volumes" Oct 30 00:36:40.031280 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:40.031243 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c26df62a-8vccg"] Oct 30 00:36:40.040055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:40.040028 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c26df62a-8vccg"] Oct 30 00:36:40.920055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:40.920021 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="405f5b62-eb46-429f-9f68-9f444812c478" path="/var/lib/kubelet/pods/405f5b62-eb46-429f-9f68-9f444812c478/volumes" Oct 30 00:36:45.932654 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:45.932615 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54b162fb-nrhl9"] Oct 30 00:36:45.938430 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:45.938407 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b162fb-nrhl9" Oct 30 00:36:45.956293 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:45.956265 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54b162fb-nrhl9"] Oct 30 00:36:45.959453 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:45.959428 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:36:45.959638 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:45.959481 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:36:45.959712 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:45.959561 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:36:45.960326 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:45.960309 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:36:46.024463 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:46.024424 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/519641a7-422a-40ec-8150-084b8572ebc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54b162fb-nrhl9\" (UID: \"519641a7-422a-40ec-8150-084b8572ebc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b162fb-nrhl9" Oct 30 00:36:46.024724 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:46.024495 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjl2f\" (UniqueName: \"kubernetes.io/projected/519641a7-422a-40ec-8150-084b8572ebc1-kube-api-access-rjl2f\") pod \"maas-api-revoke-sub-54b162fb-nrhl9\" (UID: \"519641a7-422a-40ec-8150-084b8572ebc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b162fb-nrhl9" Oct 30 00:36:46.024724 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:46.024676 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/519641a7-422a-40ec-8150-084b8572ebc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54b162fb-nrhl9\" (UID: \"519641a7-422a-40ec-8150-084b8572ebc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b162fb-nrhl9" Oct 30 00:36:46.126623 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:46.126590 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/519641a7-422a-40ec-8150-084b8572ebc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54b162fb-nrhl9\" (UID: \"519641a7-422a-40ec-8150-084b8572ebc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b162fb-nrhl9" Oct 30 00:36:46.126808 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:46.126679 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjl2f\" (UniqueName: \"kubernetes.io/projected/519641a7-422a-40ec-8150-084b8572ebc1-kube-api-access-rjl2f\") pod \"maas-api-revoke-sub-54b162fb-nrhl9\" (UID: \"519641a7-422a-40ec-8150-084b8572ebc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b162fb-nrhl9" Oct 30 00:36:46.126900 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:46.126861 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/519641a7-422a-40ec-8150-084b8572ebc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54b162fb-nrhl9\" (UID: \"519641a7-422a-40ec-8150-084b8572ebc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b162fb-nrhl9" Oct 30 00:36:46.127315 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:46.127286 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/519641a7-422a-40ec-8150-084b8572ebc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54b162fb-nrhl9\" (UID: \"519641a7-422a-40ec-8150-084b8572ebc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b162fb-nrhl9" Oct 30 00:36:46.129295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:46.129271 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/519641a7-422a-40ec-8150-084b8572ebc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54b162fb-nrhl9\" (UID: \"519641a7-422a-40ec-8150-084b8572ebc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b162fb-nrhl9" Oct 30 00:36:46.147274 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:46.147248 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjl2f\" (UniqueName: \"kubernetes.io/projected/519641a7-422a-40ec-8150-084b8572ebc1-kube-api-access-rjl2f\") pod \"maas-api-revoke-sub-54b162fb-nrhl9\" (UID: \"519641a7-422a-40ec-8150-084b8572ebc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b162fb-nrhl9" Oct 30 00:36:46.254288 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:46.254200 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b162fb-nrhl9" Oct 30 00:36:46.561473 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:36:46.561442 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod519641a7_422a_40ec_8150_084b8572ebc1.slice/crio-f2c080bf8ba8363063e2003aecce825a872005c2454333e2a7467275ceec65a6 WatchSource:0}: Error finding container f2c080bf8ba8363063e2003aecce825a872005c2454333e2a7467275ceec65a6: Status 404 returned error can't find the container with id f2c080bf8ba8363063e2003aecce825a872005c2454333e2a7467275ceec65a6 Oct 30 00:36:46.562935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:46.562897 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54b162fb-nrhl9"] Oct 30 00:36:46.829205 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:46.829177 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b162fb-nrhl9" event={"ID":"519641a7-422a-40ec-8150-084b8572ebc1","Type":"ContainerStarted","Data":"f2c080bf8ba8363063e2003aecce825a872005c2454333e2a7467275ceec65a6"} Oct 30 00:36:47.324517 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:36:47.324479 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:36:47.842667 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:47.842630 5001 generic.go:356] "Generic (PLEG): container finished" podID="519641a7-422a-40ec-8150-084b8572ebc1" containerID="7608f0e5cb43f1e2074e5fe8f4e7cb0548347b7413a172bf1e69e9fc3f871c0e" exitCode=0 Oct 30 00:36:47.842846 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:47.842680 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b162fb-nrhl9" event={"ID":"519641a7-422a-40ec-8150-084b8572ebc1","Type":"ContainerDied","Data":"7608f0e5cb43f1e2074e5fe8f4e7cb0548347b7413a172bf1e69e9fc3f871c0e"} Oct 30 00:36:49.026997 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:49.026973 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b162fb-nrhl9" Oct 30 00:36:49.096130 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:49.096089 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/519641a7-422a-40ec-8150-084b8572ebc1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/519641a7-422a-40ec-8150-084b8572ebc1-maas-api-xks-ca\") pod \"519641a7-422a-40ec-8150-084b8572ebc1\" (UID: \"519641a7-422a-40ec-8150-084b8572ebc1\") " Oct 30 00:36:49.096454 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:49.096424 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/519641a7-422a-40ec-8150-084b8572ebc1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/519641a7-422a-40ec-8150-084b8572ebc1-maas-api-openshift-ca\") pod \"519641a7-422a-40ec-8150-084b8572ebc1\" (UID: \"519641a7-422a-40ec-8150-084b8572ebc1\") " Oct 30 00:36:49.096570 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:49.096505 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/519641a7-422a-40ec-8150-084b8572ebc1-kube-api-access-rjl2f\" (UniqueName: \"kubernetes.io/projected/519641a7-422a-40ec-8150-084b8572ebc1-kube-api-access-rjl2f\") pod \"519641a7-422a-40ec-8150-084b8572ebc1\" (UID: \"519641a7-422a-40ec-8150-084b8572ebc1\") " Oct 30 00:36:49.096870 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:49.096834 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/519641a7-422a-40ec-8150-084b8572ebc1-maas-api-openshift-ca" pod "519641a7-422a-40ec-8150-084b8572ebc1" (UID: "519641a7-422a-40ec-8150-084b8572ebc1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:36:49.097280 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:49.097251 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/519641a7-422a-40ec-8150-084b8572ebc1-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:36:49.098680 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:49.098653 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/519641a7-422a-40ec-8150-084b8572ebc1-maas-api-xks-ca" pod "519641a7-422a-40ec-8150-084b8572ebc1" (UID: "519641a7-422a-40ec-8150-084b8572ebc1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:36:49.098774 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:49.098757 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/519641a7-422a-40ec-8150-084b8572ebc1-kube-api-access-rjl2f" pod "519641a7-422a-40ec-8150-084b8572ebc1" (UID: "519641a7-422a-40ec-8150-084b8572ebc1"). InnerVolumeSpecName "kube-api-access-rjl2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:36:49.199468 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:49.199434 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjl2f\" (UniqueName: \"kubernetes.io/projected/519641a7-422a-40ec-8150-084b8572ebc1-kube-api-access-rjl2f\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:36:49.199468 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:49.199467 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/519641a7-422a-40ec-8150-084b8572ebc1-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:36:49.862231 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:49.862203 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b162fb-nrhl9" Oct 30 00:36:49.862231 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:49.862230 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b162fb-nrhl9" event={"ID":"519641a7-422a-40ec-8150-084b8572ebc1","Type":"ContainerDied","Data":"f2c080bf8ba8363063e2003aecce825a872005c2454333e2a7467275ceec65a6"} Oct 30 00:36:49.862452 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:49.862277 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2c080bf8ba8363063e2003aecce825a872005c2454333e2a7467275ceec65a6" Oct 30 00:36:56.029196 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:56.029116 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3ea63d4-vmfh9"] Oct 30 00:36:56.038032 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:56.038005 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3ea63d4-vmfh9"] Oct 30 00:36:56.250394 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:56.250356 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d87c9387-8v4s4"] Oct 30 00:36:56.252270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:56.252242 5001 cpu_manager.go:414] "containerMap: removing container" podUID="519641a7-422a-40ec-8150-084b8572ebc1" containerName="revoke-keys" Oct 30 00:36:56.252449 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:56.252434 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="519641a7-422a-40ec-8150-084b8572ebc1" containerName="revoke-keys" Oct 30 00:36:56.256931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:56.252891 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="519641a7-422a-40ec-8150-084b8572ebc1" containerName="revoke-keys" Oct 30 00:36:56.262842 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:56.262820 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87c9387-8v4s4" Oct 30 00:36:56.276982 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:56.275529 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:36:56.276982 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:56.275621 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:36:56.276982 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:56.275682 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:36:56.276982 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:56.275710 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:36:56.276982 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:56.276104 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d87c9387-8v4s4"] Oct 30 00:36:56.339020 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:56.338898 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvg2j\" (UniqueName: \"kubernetes.io/projected/e0f9021e-ef29-4bda-9add-c386e7afd191-kube-api-access-pvg2j\") pod \"maas-api-revoke-sub-d87c9387-8v4s4\" (UID: \"e0f9021e-ef29-4bda-9add-c386e7afd191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87c9387-8v4s4" Oct 30 00:36:56.339171 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:56.339150 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0f9021e-ef29-4bda-9add-c386e7afd191-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d87c9387-8v4s4\" (UID: \"e0f9021e-ef29-4bda-9add-c386e7afd191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87c9387-8v4s4" Oct 30 00:36:56.339216 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:56.339181 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0f9021e-ef29-4bda-9add-c386e7afd191-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d87c9387-8v4s4\" (UID: \"e0f9021e-ef29-4bda-9add-c386e7afd191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87c9387-8v4s4" Oct 30 00:36:56.440711 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:56.440680 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvg2j\" (UniqueName: \"kubernetes.io/projected/e0f9021e-ef29-4bda-9add-c386e7afd191-kube-api-access-pvg2j\") pod \"maas-api-revoke-sub-d87c9387-8v4s4\" (UID: \"e0f9021e-ef29-4bda-9add-c386e7afd191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87c9387-8v4s4" Oct 30 00:36:56.440896 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:56.440845 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0f9021e-ef29-4bda-9add-c386e7afd191-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d87c9387-8v4s4\" (UID: \"e0f9021e-ef29-4bda-9add-c386e7afd191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87c9387-8v4s4" Oct 30 00:36:56.440896 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:56.440869 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0f9021e-ef29-4bda-9add-c386e7afd191-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d87c9387-8v4s4\" (UID: \"e0f9021e-ef29-4bda-9add-c386e7afd191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87c9387-8v4s4" Oct 30 00:36:56.441568 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:56.441545 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0f9021e-ef29-4bda-9add-c386e7afd191-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d87c9387-8v4s4\" (UID: \"e0f9021e-ef29-4bda-9add-c386e7afd191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87c9387-8v4s4" Oct 30 00:36:56.443335 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:56.443313 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0f9021e-ef29-4bda-9add-c386e7afd191-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d87c9387-8v4s4\" (UID: \"e0f9021e-ef29-4bda-9add-c386e7afd191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87c9387-8v4s4" Oct 30 00:36:56.461741 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:56.461711 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvg2j\" (UniqueName: \"kubernetes.io/projected/e0f9021e-ef29-4bda-9add-c386e7afd191-kube-api-access-pvg2j\") pod \"maas-api-revoke-sub-d87c9387-8v4s4\" (UID: \"e0f9021e-ef29-4bda-9add-c386e7afd191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87c9387-8v4s4" Oct 30 00:36:56.581378 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:56.581348 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87c9387-8v4s4" Oct 30 00:36:56.884784 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:36:56.884754 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0f9021e_ef29_4bda_9add_c386e7afd191.slice/crio-180121b951e7fbc2ca7ea18ac32f1cc582d5566f95d3cc1d0f7ff86185a6b89f WatchSource:0}: Error finding container 180121b951e7fbc2ca7ea18ac32f1cc582d5566f95d3cc1d0f7ff86185a6b89f: Status 404 returned error can't find the container with id 180121b951e7fbc2ca7ea18ac32f1cc582d5566f95d3cc1d0f7ff86185a6b89f Oct 30 00:36:56.885477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:56.885447 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d87c9387-8v4s4"] Oct 30 00:36:56.921520 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:56.921495 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa9f1f3d-4112-4ce6-bff3-c789cc809901" path="/var/lib/kubelet/pods/aa9f1f3d-4112-4ce6-bff3-c789cc809901/volumes" Oct 30 00:36:56.929358 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:56.929331 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87c9387-8v4s4" event={"ID":"e0f9021e-ef29-4bda-9add-c386e7afd191","Type":"ContainerStarted","Data":"180121b951e7fbc2ca7ea18ac32f1cc582d5566f95d3cc1d0f7ff86185a6b89f"} Oct 30 00:36:57.940209 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:57.940170 5001 generic.go:356] "Generic (PLEG): container finished" podID="e0f9021e-ef29-4bda-9add-c386e7afd191" containerID="cfe4805468d685d0638db46bc680f99ec5fdafd0e90f89119d4b730f42bc5c4f" exitCode=0 Oct 30 00:36:57.940620 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:57.940206 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87c9387-8v4s4" event={"ID":"e0f9021e-ef29-4bda-9add-c386e7afd191","Type":"ContainerDied","Data":"cfe4805468d685d0638db46bc680f99ec5fdafd0e90f89119d4b730f42bc5c4f"} Oct 30 00:36:59.107322 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:59.107292 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87c9387-8v4s4" Oct 30 00:36:59.181924 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:59.181881 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0f9021e-ef29-4bda-9add-c386e7afd191-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0f9021e-ef29-4bda-9add-c386e7afd191-maas-api-openshift-ca\") pod \"e0f9021e-ef29-4bda-9add-c386e7afd191\" (UID: \"e0f9021e-ef29-4bda-9add-c386e7afd191\") " Oct 30 00:36:59.182097 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:59.181979 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0f9021e-ef29-4bda-9add-c386e7afd191-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0f9021e-ef29-4bda-9add-c386e7afd191-maas-api-xks-ca\") pod \"e0f9021e-ef29-4bda-9add-c386e7afd191\" (UID: \"e0f9021e-ef29-4bda-9add-c386e7afd191\") " Oct 30 00:36:59.182097 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:59.182050 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0f9021e-ef29-4bda-9add-c386e7afd191-kube-api-access-pvg2j\" (UniqueName: \"kubernetes.io/projected/e0f9021e-ef29-4bda-9add-c386e7afd191-kube-api-access-pvg2j\") pod \"e0f9021e-ef29-4bda-9add-c386e7afd191\" (UID: \"e0f9021e-ef29-4bda-9add-c386e7afd191\") " Oct 30 00:36:59.182320 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:59.182291 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0f9021e-ef29-4bda-9add-c386e7afd191-maas-api-openshift-ca" pod "e0f9021e-ef29-4bda-9add-c386e7afd191" (UID: "e0f9021e-ef29-4bda-9add-c386e7afd191"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:36:59.182653 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:59.182630 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0f9021e-ef29-4bda-9add-c386e7afd191-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:36:59.184313 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:59.184285 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0f9021e-ef29-4bda-9add-c386e7afd191-kube-api-access-pvg2j" pod "e0f9021e-ef29-4bda-9add-c386e7afd191" (UID: "e0f9021e-ef29-4bda-9add-c386e7afd191"). InnerVolumeSpecName "kube-api-access-pvg2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:36:59.184423 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:59.184393 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0f9021e-ef29-4bda-9add-c386e7afd191-maas-api-xks-ca" pod "e0f9021e-ef29-4bda-9add-c386e7afd191" (UID: "e0f9021e-ef29-4bda-9add-c386e7afd191"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:36:59.285371 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:59.284146 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0f9021e-ef29-4bda-9add-c386e7afd191-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:36:59.285371 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:59.284185 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvg2j\" (UniqueName: \"kubernetes.io/projected/e0f9021e-ef29-4bda-9add-c386e7afd191-kube-api-access-pvg2j\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:36:59.959781 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:59.959750 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87c9387-8v4s4" Oct 30 00:36:59.959995 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:59.959782 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87c9387-8v4s4" event={"ID":"e0f9021e-ef29-4bda-9add-c386e7afd191","Type":"ContainerDied","Data":"180121b951e7fbc2ca7ea18ac32f1cc582d5566f95d3cc1d0f7ff86185a6b89f"} Oct 30 00:36:59.959995 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:36:59.959844 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="180121b951e7fbc2ca7ea18ac32f1cc582d5566f95d3cc1d0f7ff86185a6b89f" Oct 30 00:37:01.127522 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:01.127489 5001 scope.go:122] "RemoveContainer" containerID="d6f13f34ea1c7cccc53af9f0dfabfe52acb5d4adcdefd744dcba94f5e5057f7d" Oct 30 00:37:01.143561 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:01.143541 5001 scope.go:122] "RemoveContainer" containerID="7a56e41df38b169aae2c7005fc13191c3e49768bbac6a268b7e0be48b871d3d2" Oct 30 00:37:01.154665 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:01.154644 5001 scope.go:122] "RemoveContainer" containerID="fad4d0c3a5c2d060e257ad81dae9090f6cf3d0fe724f30903c34b44cbe5c3350" Oct 30 00:37:01.164685 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:01.164665 5001 scope.go:122] "RemoveContainer" containerID="80d2597e5cbfa1e846d7fba977afd9b7d58fc884f12475458eee81e6c09362e6" Oct 30 00:37:01.174310 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:01.174285 5001 scope.go:122] "RemoveContainer" containerID="f95650d02898872208e7c752adf838a48a21b56c427f7641368615c92d74ca13" Oct 30 00:37:07.820146 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:07.820113 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a7ac6-c9sl2"] Oct 30 00:37:07.820840 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:07.820825 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e0f9021e-ef29-4bda-9add-c386e7afd191" containerName="revoke-keys" Oct 30 00:37:07.820882 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:07.820841 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0f9021e-ef29-4bda-9add-c386e7afd191" containerName="revoke-keys" Oct 30 00:37:07.820978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:07.820966 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0f9021e-ef29-4bda-9add-c386e7afd191" containerName="revoke-keys" Oct 30 00:37:07.824497 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:07.824478 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a7ac6-c9sl2" Oct 30 00:37:07.839102 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:07.839072 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:37:07.849105 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:07.849085 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:37:07.849239 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:07.849209 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:37:07.849307 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:07.849241 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:37:07.855710 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:07.855686 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a7ac6-c9sl2"] Oct 30 00:37:07.935345 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:07.935315 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3de6f63-a674-4c76-ace4-55b3c9f2cc12-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-7a7ac6-c9sl2\" (UID: \"b3de6f63-a674-4c76-ace4-55b3c9f2cc12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a7ac6-c9sl2" Oct 30 00:37:07.935517 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:07.935387 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7zbf\" (UniqueName: \"kubernetes.io/projected/b3de6f63-a674-4c76-ace4-55b3c9f2cc12-kube-api-access-r7zbf\") pod \"maas-api-revoke-keys-e2e-worker-w1-7a7ac6-c9sl2\" (UID: \"b3de6f63-a674-4c76-ace4-55b3c9f2cc12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a7ac6-c9sl2" Oct 30 00:37:07.935517 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:07.935470 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3de6f63-a674-4c76-ace4-55b3c9f2cc12-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-7a7ac6-c9sl2\" (UID: \"b3de6f63-a674-4c76-ace4-55b3c9f2cc12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a7ac6-c9sl2" Oct 30 00:37:08.037100 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:08.037068 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3de6f63-a674-4c76-ace4-55b3c9f2cc12-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-7a7ac6-c9sl2\" (UID: \"b3de6f63-a674-4c76-ace4-55b3c9f2cc12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a7ac6-c9sl2" Oct 30 00:37:08.037312 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:08.037298 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3de6f63-a674-4c76-ace4-55b3c9f2cc12-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-7a7ac6-c9sl2\" (UID: \"b3de6f63-a674-4c76-ace4-55b3c9f2cc12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a7ac6-c9sl2" Oct 30 00:37:08.037384 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:08.037348 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7zbf\" (UniqueName: \"kubernetes.io/projected/b3de6f63-a674-4c76-ace4-55b3c9f2cc12-kube-api-access-r7zbf\") pod \"maas-api-revoke-keys-e2e-worker-w1-7a7ac6-c9sl2\" (UID: \"b3de6f63-a674-4c76-ace4-55b3c9f2cc12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a7ac6-c9sl2" Oct 30 00:37:08.037795 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:08.037766 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3de6f63-a674-4c76-ace4-55b3c9f2cc12-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-7a7ac6-c9sl2\" (UID: \"b3de6f63-a674-4c76-ace4-55b3c9f2cc12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a7ac6-c9sl2" Oct 30 00:37:08.039822 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:08.039792 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3de6f63-a674-4c76-ace4-55b3c9f2cc12-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-7a7ac6-c9sl2\" (UID: \"b3de6f63-a674-4c76-ace4-55b3c9f2cc12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a7ac6-c9sl2" Oct 30 00:37:08.058221 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:08.058199 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7zbf\" (UniqueName: \"kubernetes.io/projected/b3de6f63-a674-4c76-ace4-55b3c9f2cc12-kube-api-access-r7zbf\") pod \"maas-api-revoke-keys-e2e-worker-w1-7a7ac6-c9sl2\" (UID: \"b3de6f63-a674-4c76-ace4-55b3c9f2cc12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a7ac6-c9sl2" Oct 30 00:37:08.138824 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:08.138738 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a7ac6-c9sl2" Oct 30 00:37:08.458617 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:37:08.458582 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3de6f63_a674_4c76_ace4_55b3c9f2cc12.slice/crio-2aa039cf4184bdc8dd3131bcdabeaa7afe92e666964aac3402bc732d5c44e6c9 WatchSource:0}: Error finding container 2aa039cf4184bdc8dd3131bcdabeaa7afe92e666964aac3402bc732d5c44e6c9: Status 404 returned error can't find the container with id 2aa039cf4184bdc8dd3131bcdabeaa7afe92e666964aac3402bc732d5c44e6c9 Oct 30 00:37:08.460348 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:08.460321 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a7ac6-c9sl2"] Oct 30 00:37:09.029424 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:09.029389 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac8930ef-x4xqc"] Oct 30 00:37:09.039964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:09.039902 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac8930ef-x4xqc"] Oct 30 00:37:09.045827 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:09.045793 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a7ac6-c9sl2" event={"ID":"b3de6f63-a674-4c76-ace4-55b3c9f2cc12","Type":"ContainerStarted","Data":"8023c49f41499410c3850b0c5d653cef27c30f54f29c7f8ce3994df081aaafd5"} Oct 30 00:37:09.046018 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:09.045852 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a7ac6-c9sl2" event={"ID":"b3de6f63-a674-4c76-ace4-55b3c9f2cc12","Type":"ContainerStarted","Data":"2aa039cf4184bdc8dd3131bcdabeaa7afe92e666964aac3402bc732d5c44e6c9"} Oct 30 00:37:09.080314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:09.080267 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a7ac6-c9sl2" podStartSLOduration=2.080255904 podStartE2EDuration="2.080255904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:37:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:37:09.079992604 +0000 UTC m=+1944.285493785" watchObservedRunningTime="2026-10-30 00:37:09.080255904 +0000 UTC m=+1944.285757085" Oct 30 00:37:10.185959 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:10.185234 5001 generic.go:356] "Generic (PLEG): container finished" podID="b3de6f63-a674-4c76-ace4-55b3c9f2cc12" containerID="8023c49f41499410c3850b0c5d653cef27c30f54f29c7f8ce3994df081aaafd5" exitCode=0 Oct 30 00:37:10.185959 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:10.185364 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a7ac6-c9sl2" event={"ID":"b3de6f63-a674-4c76-ace4-55b3c9f2cc12","Type":"ContainerDied","Data":"8023c49f41499410c3850b0c5d653cef27c30f54f29c7f8ce3994df081aaafd5"} Oct 30 00:37:10.964944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:10.962563 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b" path="/var/lib/kubelet/pods/46bbfb19-c05d-4ce5-b5ee-9f47e8618b9b/volumes" Oct 30 00:37:11.463356 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:11.463334 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a7ac6-c9sl2" Oct 30 00:37:11.584331 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:11.584299 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3de6f63-a674-4c76-ace4-55b3c9f2cc12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3de6f63-a674-4c76-ace4-55b3c9f2cc12-maas-api-xks-ca\") pod \"b3de6f63-a674-4c76-ace4-55b3c9f2cc12\" (UID: \"b3de6f63-a674-4c76-ace4-55b3c9f2cc12\") " Oct 30 00:37:11.584515 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:11.584361 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3de6f63-a674-4c76-ace4-55b3c9f2cc12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3de6f63-a674-4c76-ace4-55b3c9f2cc12-maas-api-openshift-ca\") pod \"b3de6f63-a674-4c76-ace4-55b3c9f2cc12\" (UID: \"b3de6f63-a674-4c76-ace4-55b3c9f2cc12\") " Oct 30 00:37:11.584590 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:11.584563 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3de6f63-a674-4c76-ace4-55b3c9f2cc12-kube-api-access-r7zbf\" (UniqueName: \"kubernetes.io/projected/b3de6f63-a674-4c76-ace4-55b3c9f2cc12-kube-api-access-r7zbf\") pod \"b3de6f63-a674-4c76-ace4-55b3c9f2cc12\" (UID: \"b3de6f63-a674-4c76-ace4-55b3c9f2cc12\") " Oct 30 00:37:11.584777 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:11.584747 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3de6f63-a674-4c76-ace4-55b3c9f2cc12-maas-api-openshift-ca" pod "b3de6f63-a674-4c76-ace4-55b3c9f2cc12" (UID: "b3de6f63-a674-4c76-ace4-55b3c9f2cc12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:37:11.585259 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:11.585242 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3de6f63-a674-4c76-ace4-55b3c9f2cc12-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:11.586542 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:11.586520 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3de6f63-a674-4c76-ace4-55b3c9f2cc12-kube-api-access-r7zbf" pod "b3de6f63-a674-4c76-ace4-55b3c9f2cc12" (UID: "b3de6f63-a674-4c76-ace4-55b3c9f2cc12"). InnerVolumeSpecName "kube-api-access-r7zbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:11.586612 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:11.586555 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3de6f63-a674-4c76-ace4-55b3c9f2cc12-maas-api-xks-ca" pod "b3de6f63-a674-4c76-ace4-55b3c9f2cc12" (UID: "b3de6f63-a674-4c76-ace4-55b3c9f2cc12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:37:11.686706 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:11.686674 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7zbf\" (UniqueName: \"kubernetes.io/projected/b3de6f63-a674-4c76-ace4-55b3c9f2cc12-kube-api-access-r7zbf\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:11.686706 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:11.686705 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3de6f63-a674-4c76-ace4-55b3c9f2cc12-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:12.227206 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:12.227176 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a7ac6-c9sl2" Oct 30 00:37:12.227380 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:12.227211 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a7ac6-c9sl2" event={"ID":"b3de6f63-a674-4c76-ace4-55b3c9f2cc12","Type":"ContainerDied","Data":"2aa039cf4184bdc8dd3131bcdabeaa7afe92e666964aac3402bc732d5c44e6c9"} Oct 30 00:37:12.227380 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:12.227260 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2aa039cf4184bdc8dd3131bcdabeaa7afe92e666964aac3402bc732d5c44e6c9" Oct 30 00:37:12.534243 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:12.534211 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc"] Oct 30 00:37:12.545332 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:12.545305 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh"] Oct 30 00:37:12.550226 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:12.550202 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq"] Oct 30 00:37:13.353366 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:13.353304 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34e401f2-pl7wk"] Oct 30 00:37:13.354784 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:13.354754 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b3de6f63-a674-4c76-ace4-55b3c9f2cc12" containerName="revoke-keys" Oct 30 00:37:13.354986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:13.354969 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3de6f63-a674-4c76-ace4-55b3c9f2cc12" containerName="revoke-keys" Oct 30 00:37:13.355274 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:13.355261 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3de6f63-a674-4c76-ace4-55b3c9f2cc12" containerName="revoke-keys" Oct 30 00:37:13.381248 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:13.381211 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e401f2-pl7wk" Oct 30 00:37:13.403527 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:13.403489 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:37:13.422168 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:13.422103 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34e401f2-pl7wk"] Oct 30 00:37:13.427463 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:13.426495 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34e401f2-pl7wk\" (UID: \"c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e401f2-pl7wk" Oct 30 00:37:13.427463 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:13.426687 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tcss\" (UniqueName: \"kubernetes.io/projected/c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad-kube-api-access-8tcss\") pod \"maas-api-revoke-sub-34e401f2-pl7wk\" (UID: \"c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e401f2-pl7wk" Oct 30 00:37:13.427463 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:13.426811 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34e401f2-pl7wk\" (UID: \"c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e401f2-pl7wk" Oct 30 00:37:13.435971 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:13.433392 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:37:13.435971 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:13.433514 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:37:13.435971 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:13.433604 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:37:13.527671 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:13.527637 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34e401f2-pl7wk\" (UID: \"c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e401f2-pl7wk" Oct 30 00:37:13.527845 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:13.527771 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tcss\" (UniqueName: \"kubernetes.io/projected/c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad-kube-api-access-8tcss\") pod \"maas-api-revoke-sub-34e401f2-pl7wk\" (UID: \"c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e401f2-pl7wk" Oct 30 00:37:13.527924 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:13.527872 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34e401f2-pl7wk\" (UID: \"c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e401f2-pl7wk" Oct 30 00:37:13.528499 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:13.528464 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34e401f2-pl7wk\" (UID: \"c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e401f2-pl7wk" Oct 30 00:37:13.530001 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:13.529982 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34e401f2-pl7wk\" (UID: \"c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e401f2-pl7wk" Oct 30 00:37:13.548152 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:13.548131 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tcss\" (UniqueName: \"kubernetes.io/projected/c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad-kube-api-access-8tcss\") pod \"maas-api-revoke-sub-34e401f2-pl7wk\" (UID: \"c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e401f2-pl7wk" Oct 30 00:37:13.713092 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:13.713011 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5f7c4cb-rtqsc"] Oct 30 00:37:13.717871 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:13.717845 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f7c4cb-rtqsc" Oct 30 00:37:13.738562 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:13.738530 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5f7c4cb-rtqsc"] Oct 30 00:37:13.744291 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:13.744268 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e401f2-pl7wk" Oct 30 00:37:13.831648 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:13.831612 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbhzx\" (UniqueName: \"kubernetes.io/projected/14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078-kube-api-access-hbhzx\") pod \"maas-api-revoke-sub-f5f7c4cb-rtqsc\" (UID: \"14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f7c4cb-rtqsc" Oct 30 00:37:13.831980 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:13.831954 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5f7c4cb-rtqsc\" (UID: \"14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f7c4cb-rtqsc" Oct 30 00:37:13.832090 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:13.832067 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5f7c4cb-rtqsc\" (UID: \"14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f7c4cb-rtqsc" Oct 30 00:37:13.933409 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:13.933373 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbhzx\" (UniqueName: \"kubernetes.io/projected/14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078-kube-api-access-hbhzx\") pod \"maas-api-revoke-sub-f5f7c4cb-rtqsc\" (UID: \"14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f7c4cb-rtqsc" Oct 30 00:37:13.933595 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:13.933560 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5f7c4cb-rtqsc\" (UID: \"14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f7c4cb-rtqsc" Oct 30 00:37:13.933681 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:13.933618 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5f7c4cb-rtqsc\" (UID: \"14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f7c4cb-rtqsc" Oct 30 00:37:13.934295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:13.934275 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5f7c4cb-rtqsc\" (UID: \"14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f7c4cb-rtqsc" Oct 30 00:37:13.935971 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:13.935954 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5f7c4cb-rtqsc\" (UID: \"14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f7c4cb-rtqsc" Oct 30 00:37:13.953947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:13.953925 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbhzx\" (UniqueName: \"kubernetes.io/projected/14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078-kube-api-access-hbhzx\") pod \"maas-api-revoke-sub-f5f7c4cb-rtqsc\" (UID: \"14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f7c4cb-rtqsc" Oct 30 00:37:14.033824 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:14.033797 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f7c4cb-rtqsc" Oct 30 00:37:14.060045 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:37:14.060020 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3efe694_3bc9_4e6e_9e3a_99ffd0dd7bad.slice/crio-ce7efc6e003341fbfedc7a9d801ce397a7abe4f649a2b0a1a21d18b757c33fa3 WatchSource:0}: Error finding container ce7efc6e003341fbfedc7a9d801ce397a7abe4f649a2b0a1a21d18b757c33fa3: Status 404 returned error can't find the container with id ce7efc6e003341fbfedc7a9d801ce397a7abe4f649a2b0a1a21d18b757c33fa3 Oct 30 00:37:14.061397 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:14.061360 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34e401f2-pl7wk"] Oct 30 00:37:14.255353 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:14.255305 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e401f2-pl7wk" event={"ID":"c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad","Type":"ContainerStarted","Data":"ce7efc6e003341fbfedc7a9d801ce397a7abe4f649a2b0a1a21d18b757c33fa3"} Oct 30 00:37:14.347506 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:37:14.347474 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d68ed9_c3b2_48f4_8edb_0d4c2c5f0078.slice/crio-3e35e9e82eb11399a89e0ebd4d32d1e81272a1a2b0aba36d3c5b1861dbb07b90 WatchSource:0}: Error finding container 3e35e9e82eb11399a89e0ebd4d32d1e81272a1a2b0aba36d3c5b1861dbb07b90: Status 404 returned error can't find the container with id 3e35e9e82eb11399a89e0ebd4d32d1e81272a1a2b0aba36d3c5b1861dbb07b90 Oct 30 00:37:14.348712 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:14.348686 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5f7c4cb-rtqsc"] Oct 30 00:37:15.266202 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:15.266168 5001 generic.go:356] "Generic (PLEG): container finished" podID="c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad" containerID="f3e4c5eaa9d79dd38029d42b5abf58c23c4b309009a3df3b3e0553460180de3b" exitCode=0 Oct 30 00:37:15.266592 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:15.266242 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e401f2-pl7wk" event={"ID":"c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad","Type":"ContainerDied","Data":"f3e4c5eaa9d79dd38029d42b5abf58c23c4b309009a3df3b3e0553460180de3b"} Oct 30 00:37:15.267850 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:15.267823 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f7c4cb-rtqsc" event={"ID":"14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078","Type":"ContainerStarted","Data":"d8e691193e2ad4561efcd3aa4aeebd573c5519e7a3545dcc52b56dd3fb7c1a1c"} Oct 30 00:37:15.267999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:15.267874 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f7c4cb-rtqsc" event={"ID":"14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078","Type":"ContainerStarted","Data":"3e35e9e82eb11399a89e0ebd4d32d1e81272a1a2b0aba36d3c5b1861dbb07b90"} Oct 30 00:37:15.334986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:15.334934 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f7c4cb-rtqsc" podStartSLOduration=2.33490215 podStartE2EDuration="2.33490215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:37:15.333834659 +0000 UTC m=+1950.539335841" watchObservedRunningTime="2026-10-30 00:37:15.33490215 +0000 UTC m=+1950.540403328" Oct 30 00:37:16.029034 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:16.028998 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91e3e499-kj5d7"] Oct 30 00:37:16.038425 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:16.038398 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91e3e499-kj5d7"] Oct 30 00:37:16.321998 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:16.321053 5001 generic.go:356] "Generic (PLEG): container finished" podID="14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078" containerID="d8e691193e2ad4561efcd3aa4aeebd573c5519e7a3545dcc52b56dd3fb7c1a1c" exitCode=0 Oct 30 00:37:16.322667 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:16.321194 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f7c4cb-rtqsc" event={"ID":"14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078","Type":"ContainerDied","Data":"d8e691193e2ad4561efcd3aa4aeebd573c5519e7a3545dcc52b56dd3fb7c1a1c"} Oct 30 00:37:16.529665 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:16.529639 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e401f2-pl7wk" Oct 30 00:37:16.592985 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:16.592888 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad-kube-api-access-8tcss\" (UniqueName: \"kubernetes.io/projected/c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad-kube-api-access-8tcss\") pod \"c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad\" (UID: \"c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad\") " Oct 30 00:37:16.593139 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:16.593127 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad-maas-api-xks-ca\") pod \"c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad\" (UID: \"c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad\") " Oct 30 00:37:16.593199 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:16.593156 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad-maas-api-openshift-ca\") pod \"c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad\" (UID: \"c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad\") " Oct 30 00:37:16.593602 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:16.593575 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad-maas-api-openshift-ca" pod "c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad" (UID: "c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:37:16.593757 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:16.593742 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:16.595096 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:16.595073 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad-kube-api-access-8tcss" pod "c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad" (UID: "c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad"). InnerVolumeSpecName "kube-api-access-8tcss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:16.595415 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:16.595388 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad-maas-api-xks-ca" pod "c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad" (UID: "c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:37:16.695571 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:16.695535 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:16.695571 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:16.695565 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tcss\" (UniqueName: \"kubernetes.io/projected/c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad-kube-api-access-8tcss\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:16.920350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:16.920274 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30c22e7c-2be6-4b50-a71e-928adbe4196f" path="/var/lib/kubelet/pods/30c22e7c-2be6-4b50-a71e-928adbe4196f/volumes" Oct 30 00:37:17.329588 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:37:17.329548 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:37:17.335101 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:17.335076 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e401f2-pl7wk" Oct 30 00:37:17.335247 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:17.335112 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e401f2-pl7wk" event={"ID":"c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad","Type":"ContainerDied","Data":"ce7efc6e003341fbfedc7a9d801ce397a7abe4f649a2b0a1a21d18b757c33fa3"} Oct 30 00:37:17.335247 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:17.335168 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce7efc6e003341fbfedc7a9d801ce397a7abe4f649a2b0a1a21d18b757c33fa3" Oct 30 00:37:17.513387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:17.513364 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f7c4cb-rtqsc" Oct 30 00:37:17.608822 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:17.608727 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078-maas-api-xks-ca\") pod \"14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078\" (UID: \"14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078\") " Oct 30 00:37:17.609007 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:17.608857 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078-maas-api-openshift-ca\") pod \"14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078\" (UID: \"14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078\") " Oct 30 00:37:17.609149 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:17.609124 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078-kube-api-access-hbhzx\" (UniqueName: \"kubernetes.io/projected/14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078-kube-api-access-hbhzx\") pod \"14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078\" (UID: \"14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078\") " Oct 30 00:37:17.609280 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:17.609257 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078-maas-api-openshift-ca" pod "14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078" (UID: "14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:37:17.609941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:17.609879 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:17.611046 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:17.611020 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078-maas-api-xks-ca" pod "14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078" (UID: "14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:37:17.611568 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:17.611543 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078-kube-api-access-hbhzx" pod "14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078" (UID: "14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078"). InnerVolumeSpecName "kube-api-access-hbhzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:17.711286 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:17.711250 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbhzx\" (UniqueName: \"kubernetes.io/projected/14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078-kube-api-access-hbhzx\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:17.711286 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:17.711281 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:18.272624 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:18.272582 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7a7ac6-75b958fbc4-dqgk7"] Oct 30 00:37:18.306250 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:18.306219 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-7a7ac6-68f8c58595-vb58b"] Oct 30 00:37:18.346885 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:18.346849 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f7c4cb-rtqsc" event={"ID":"14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078","Type":"ContainerDied","Data":"3e35e9e82eb11399a89e0ebd4d32d1e81272a1a2b0aba36d3c5b1861dbb07b90"} Oct 30 00:37:18.347416 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:18.346942 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e35e9e82eb11399a89e0ebd4d32d1e81272a1a2b0aba36d3c5b1861dbb07b90" Oct 30 00:37:18.347416 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:18.346859 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f7c4cb-rtqsc" Oct 30 00:37:23.273354 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:23.273297 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a7ac6-75b958fbc4-dqgk7" podUID="30bf357c-11ff-4fc6-bcfe-c776a3370b9f" containerName="payload-processing" containerID="cri-o://f2583b68cc5f20b95025592720f04fabb73f3a6d76b0377ba09a08df4a935a5d" gracePeriod=25 Oct 30 00:37:23.307389 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:23.307346 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a7ac6-68f8c58595-vb58b" podUID="776a2d19-ec0d-4868-ac0c-e22a7c4b6de0" containerName="payload-pre-processing" containerID="cri-o://14c8a467de7b1d26a1a5858f03e940d85a951858ef045595a7184ff2c77a57d8" gracePeriod=25 Oct 30 00:37:23.659210 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:23.659128 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e58d8ab-cxfng"] Oct 30 00:37:23.659827 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:23.659810 5001 cpu_manager.go:414] "containerMap: removing container" podUID="14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078" containerName="revoke-keys" Oct 30 00:37:23.659827 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:23.659825 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078" containerName="revoke-keys" Oct 30 00:37:23.660012 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:23.659844 5001 cpu_manager.go:414] "containerMap: removing container" podUID="c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad" containerName="revoke-keys" Oct 30 00:37:23.660012 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:23.659850 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad" containerName="revoke-keys" Oct 30 00:37:23.660012 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:23.660003 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078" containerName="revoke-keys" Oct 30 00:37:23.660012 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:23.660013 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad" containerName="revoke-keys" Oct 30 00:37:23.663490 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:23.663470 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e58d8ab-cxfng" Oct 30 00:37:23.678130 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:23.678111 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:37:23.685623 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:23.685600 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e58d8ab-cxfng"] Oct 30 00:37:23.688110 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:23.688089 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:37:23.688233 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:23.688096 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:37:23.688332 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:23.688314 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:37:23.822046 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:23.822007 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crtxq\" (UniqueName: \"kubernetes.io/projected/0989dd93-7d85-4178-b9b0-48c87f0970a9-kube-api-access-crtxq\") pod \"maas-api-revoke-sub-7e58d8ab-cxfng\" (UID: \"0989dd93-7d85-4178-b9b0-48c87f0970a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e58d8ab-cxfng" Oct 30 00:37:23.822228 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:23.822129 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0989dd93-7d85-4178-b9b0-48c87f0970a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e58d8ab-cxfng\" (UID: \"0989dd93-7d85-4178-b9b0-48c87f0970a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e58d8ab-cxfng" Oct 30 00:37:23.822228 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:23.822183 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0989dd93-7d85-4178-b9b0-48c87f0970a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e58d8ab-cxfng\" (UID: \"0989dd93-7d85-4178-b9b0-48c87f0970a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e58d8ab-cxfng" Oct 30 00:37:23.873997 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:23.873965 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85d234d0-r994g"] Oct 30 00:37:23.878792 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:23.878772 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d234d0-r994g" Oct 30 00:37:23.896293 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:23.896254 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30713353-4nrqg"] Oct 30 00:37:23.901840 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:23.901818 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30713353-4nrqg" Oct 30 00:37:23.903002 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:23.902976 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85d234d0-r994g"] Oct 30 00:37:23.924302 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:23.924278 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crtxq\" (UniqueName: \"kubernetes.io/projected/0989dd93-7d85-4178-b9b0-48c87f0970a9-kube-api-access-crtxq\") pod \"maas-api-revoke-sub-7e58d8ab-cxfng\" (UID: \"0989dd93-7d85-4178-b9b0-48c87f0970a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e58d8ab-cxfng" Oct 30 00:37:23.924407 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:23.924382 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0989dd93-7d85-4178-b9b0-48c87f0970a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e58d8ab-cxfng\" (UID: \"0989dd93-7d85-4178-b9b0-48c87f0970a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e58d8ab-cxfng" Oct 30 00:37:23.924449 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:23.924410 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0989dd93-7d85-4178-b9b0-48c87f0970a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e58d8ab-cxfng\" (UID: \"0989dd93-7d85-4178-b9b0-48c87f0970a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e58d8ab-cxfng" Oct 30 00:37:23.925726 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:23.925699 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0989dd93-7d85-4178-b9b0-48c87f0970a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e58d8ab-cxfng\" (UID: \"0989dd93-7d85-4178-b9b0-48c87f0970a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e58d8ab-cxfng" Oct 30 00:37:23.927209 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:23.927185 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0989dd93-7d85-4178-b9b0-48c87f0970a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e58d8ab-cxfng\" (UID: \"0989dd93-7d85-4178-b9b0-48c87f0970a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e58d8ab-cxfng" Oct 30 00:37:23.927716 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:23.927697 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30713353-4nrqg"] Oct 30 00:37:23.949060 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:23.949032 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crtxq\" (UniqueName: \"kubernetes.io/projected/0989dd93-7d85-4178-b9b0-48c87f0970a9-kube-api-access-crtxq\") pod \"maas-api-revoke-sub-7e58d8ab-cxfng\" (UID: \"0989dd93-7d85-4178-b9b0-48c87f0970a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e58d8ab-cxfng" Oct 30 00:37:23.978089 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:23.978067 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e58d8ab-cxfng" Oct 30 00:37:24.027355 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.027320 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czngw\" (UniqueName: \"kubernetes.io/projected/f86ca908-fc4f-4c3a-a519-5d8f928576e3-kube-api-access-czngw\") pod \"maas-api-revoke-sub-85d234d0-r994g\" (UID: \"f86ca908-fc4f-4c3a-a519-5d8f928576e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d234d0-r994g" Oct 30 00:37:24.027525 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.027376 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da718472-0862-4f25-8043-057c7fb59d17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30713353-4nrqg\" (UID: \"da718472-0862-4f25-8043-057c7fb59d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30713353-4nrqg" Oct 30 00:37:24.027525 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.027507 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f86ca908-fc4f-4c3a-a519-5d8f928576e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85d234d0-r994g\" (UID: \"f86ca908-fc4f-4c3a-a519-5d8f928576e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d234d0-r994g" Oct 30 00:37:24.027651 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.027565 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da718472-0862-4f25-8043-057c7fb59d17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30713353-4nrqg\" (UID: \"da718472-0862-4f25-8043-057c7fb59d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30713353-4nrqg" Oct 30 00:37:24.027749 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.027727 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntfjk\" (UniqueName: \"kubernetes.io/projected/da718472-0862-4f25-8043-057c7fb59d17-kube-api-access-ntfjk\") pod \"maas-api-revoke-sub-30713353-4nrqg\" (UID: \"da718472-0862-4f25-8043-057c7fb59d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30713353-4nrqg" Oct 30 00:37:24.027845 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.027829 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f86ca908-fc4f-4c3a-a519-5d8f928576e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85d234d0-r994g\" (UID: \"f86ca908-fc4f-4c3a-a519-5d8f928576e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d234d0-r994g" Oct 30 00:37:24.115067 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.115026 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa4e22fb-7kkcm"] Oct 30 00:37:24.121514 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.121488 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4e22fb-7kkcm" Oct 30 00:37:24.130594 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.129523 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntfjk\" (UniqueName: \"kubernetes.io/projected/da718472-0862-4f25-8043-057c7fb59d17-kube-api-access-ntfjk\") pod \"maas-api-revoke-sub-30713353-4nrqg\" (UID: \"da718472-0862-4f25-8043-057c7fb59d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30713353-4nrqg" Oct 30 00:37:24.130594 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.129644 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f86ca908-fc4f-4c3a-a519-5d8f928576e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85d234d0-r994g\" (UID: \"f86ca908-fc4f-4c3a-a519-5d8f928576e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d234d0-r994g" Oct 30 00:37:24.130594 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.129786 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czngw\" (UniqueName: \"kubernetes.io/projected/f86ca908-fc4f-4c3a-a519-5d8f928576e3-kube-api-access-czngw\") pod \"maas-api-revoke-sub-85d234d0-r994g\" (UID: \"f86ca908-fc4f-4c3a-a519-5d8f928576e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d234d0-r994g" Oct 30 00:37:24.130594 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.129829 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da718472-0862-4f25-8043-057c7fb59d17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30713353-4nrqg\" (UID: \"da718472-0862-4f25-8043-057c7fb59d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30713353-4nrqg" Oct 30 00:37:24.130594 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.129867 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f86ca908-fc4f-4c3a-a519-5d8f928576e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85d234d0-r994g\" (UID: \"f86ca908-fc4f-4c3a-a519-5d8f928576e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d234d0-r994g" Oct 30 00:37:24.130594 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.129948 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da718472-0862-4f25-8043-057c7fb59d17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30713353-4nrqg\" (UID: \"da718472-0862-4f25-8043-057c7fb59d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30713353-4nrqg" Oct 30 00:37:24.130594 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.130530 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da718472-0862-4f25-8043-057c7fb59d17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30713353-4nrqg\" (UID: \"da718472-0862-4f25-8043-057c7fb59d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30713353-4nrqg" Oct 30 00:37:24.131746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.131698 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f86ca908-fc4f-4c3a-a519-5d8f928576e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85d234d0-r994g\" (UID: \"f86ca908-fc4f-4c3a-a519-5d8f928576e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d234d0-r994g" Oct 30 00:37:24.133423 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.133398 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da718472-0862-4f25-8043-057c7fb59d17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30713353-4nrqg\" (UID: \"da718472-0862-4f25-8043-057c7fb59d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30713353-4nrqg" Oct 30 00:37:24.134628 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.134604 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f86ca908-fc4f-4c3a-a519-5d8f928576e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85d234d0-r994g\" (UID: \"f86ca908-fc4f-4c3a-a519-5d8f928576e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d234d0-r994g" Oct 30 00:37:24.144025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.144000 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa4e22fb-7kkcm"] Oct 30 00:37:24.151429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.151399 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntfjk\" (UniqueName: \"kubernetes.io/projected/da718472-0862-4f25-8043-057c7fb59d17-kube-api-access-ntfjk\") pod \"maas-api-revoke-sub-30713353-4nrqg\" (UID: \"da718472-0862-4f25-8043-057c7fb59d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30713353-4nrqg" Oct 30 00:37:24.152877 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.152857 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czngw\" (UniqueName: \"kubernetes.io/projected/f86ca908-fc4f-4c3a-a519-5d8f928576e3-kube-api-access-czngw\") pod \"maas-api-revoke-sub-85d234d0-r994g\" (UID: \"f86ca908-fc4f-4c3a-a519-5d8f928576e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d234d0-r994g" Oct 30 00:37:24.194932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.194848 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d234d0-r994g" Oct 30 00:37:24.218633 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.218606 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30713353-4nrqg" Oct 30 00:37:24.234498 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.234245 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d51bbbf9-d605-49bb-ae09-426c7a5622c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa4e22fb-7kkcm\" (UID: \"d51bbbf9-d605-49bb-ae09-426c7a5622c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4e22fb-7kkcm" Oct 30 00:37:24.234498 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.234308 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrw6b\" (UniqueName: \"kubernetes.io/projected/d51bbbf9-d605-49bb-ae09-426c7a5622c0-kube-api-access-wrw6b\") pod \"maas-api-revoke-sub-aa4e22fb-7kkcm\" (UID: \"d51bbbf9-d605-49bb-ae09-426c7a5622c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4e22fb-7kkcm" Oct 30 00:37:24.234701 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.234574 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d51bbbf9-d605-49bb-ae09-426c7a5622c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa4e22fb-7kkcm\" (UID: \"d51bbbf9-d605-49bb-ae09-426c7a5622c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4e22fb-7kkcm" Oct 30 00:37:24.293223 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:37:24.293174 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0989dd93_7d85_4178_b9b0_48c87f0970a9.slice/crio-2c1071a874229823c9fd3f76288c10b9e6691deff962f1e501d684febc63f759 WatchSource:0}: Error finding container 2c1071a874229823c9fd3f76288c10b9e6691deff962f1e501d684febc63f759: Status 404 returned error can't find the container with id 2c1071a874229823c9fd3f76288c10b9e6691deff962f1e501d684febc63f759 Oct 30 00:37:24.297308 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.297282 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e58d8ab-cxfng"] Oct 30 00:37:24.357944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.357379 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d51bbbf9-d605-49bb-ae09-426c7a5622c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa4e22fb-7kkcm\" (UID: \"d51bbbf9-d605-49bb-ae09-426c7a5622c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4e22fb-7kkcm" Oct 30 00:37:24.373713 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.359494 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d51bbbf9-d605-49bb-ae09-426c7a5622c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa4e22fb-7kkcm\" (UID: \"d51bbbf9-d605-49bb-ae09-426c7a5622c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4e22fb-7kkcm" Oct 30 00:37:24.373713 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.359552 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrw6b\" (UniqueName: \"kubernetes.io/projected/d51bbbf9-d605-49bb-ae09-426c7a5622c0-kube-api-access-wrw6b\") pod \"maas-api-revoke-sub-aa4e22fb-7kkcm\" (UID: \"d51bbbf9-d605-49bb-ae09-426c7a5622c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4e22fb-7kkcm" Oct 30 00:37:24.373713 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.368974 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d51bbbf9-d605-49bb-ae09-426c7a5622c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa4e22fb-7kkcm\" (UID: \"d51bbbf9-d605-49bb-ae09-426c7a5622c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4e22fb-7kkcm" Oct 30 00:37:24.377880 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.377846 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d51bbbf9-d605-49bb-ae09-426c7a5622c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa4e22fb-7kkcm\" (UID: \"d51bbbf9-d605-49bb-ae09-426c7a5622c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4e22fb-7kkcm" Oct 30 00:37:24.423399 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.423332 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrw6b\" (UniqueName: \"kubernetes.io/projected/d51bbbf9-d605-49bb-ae09-426c7a5622c0-kube-api-access-wrw6b\") pod \"maas-api-revoke-sub-aa4e22fb-7kkcm\" (UID: \"d51bbbf9-d605-49bb-ae09-426c7a5622c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4e22fb-7kkcm" Oct 30 00:37:24.431662 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.431630 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e58d8ab-cxfng" event={"ID":"0989dd93-7d85-4178-b9b0-48c87f0970a9","Type":"ContainerStarted","Data":"2c1071a874229823c9fd3f76288c10b9e6691deff962f1e501d684febc63f759"} Oct 30 00:37:24.461577 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.461545 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4e22fb-7kkcm" Oct 30 00:37:24.508198 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:37:24.508153 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf86ca908_fc4f_4c3a_a519_5d8f928576e3.slice/crio-4f9d32cc3259909de9a045008480037498d23f558f7200dcc8d2d239511e7e4d WatchSource:0}: Error finding container 4f9d32cc3259909de9a045008480037498d23f558f7200dcc8d2d239511e7e4d: Status 404 returned error can't find the container with id 4f9d32cc3259909de9a045008480037498d23f558f7200dcc8d2d239511e7e4d Oct 30 00:37:24.509443 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.509419 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85d234d0-r994g"] Oct 30 00:37:24.553230 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:37:24.553201 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda718472_0862_4f25_8043_057c7fb59d17.slice/crio-85a7080b45967429ee0cb0163543e830b69e364479f3d17cb6c4ce23e30f9276 WatchSource:0}: Error finding container 85a7080b45967429ee0cb0163543e830b69e364479f3d17cb6c4ce23e30f9276: Status 404 returned error can't find the container with id 85a7080b45967429ee0cb0163543e830b69e364479f3d17cb6c4ce23e30f9276 Oct 30 00:37:24.557531 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.557506 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30713353-4nrqg"] Oct 30 00:37:24.776160 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:37:24.776120 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd51bbbf9_d605_49bb_ae09_426c7a5622c0.slice/crio-622bc0249e2b848a0e430d9ec5aa4abf2f363c87f477965e9c1649f1e9d7347c WatchSource:0}: Error finding container 622bc0249e2b848a0e430d9ec5aa4abf2f363c87f477965e9c1649f1e9d7347c: Status 404 returned error can't find the container with id 622bc0249e2b848a0e430d9ec5aa4abf2f363c87f477965e9c1649f1e9d7347c Oct 30 00:37:24.777719 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:24.777689 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa4e22fb-7kkcm"] Oct 30 00:37:25.444583 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:25.444543 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4e22fb-7kkcm" event={"ID":"d51bbbf9-d605-49bb-ae09-426c7a5622c0","Type":"ContainerStarted","Data":"f08ac31a16523fd66c89e010a6253beabaa9f4feca8179c4677ac694bbb7dc5b"} Oct 30 00:37:25.445023 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:25.444616 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4e22fb-7kkcm" event={"ID":"d51bbbf9-d605-49bb-ae09-426c7a5622c0","Type":"ContainerStarted","Data":"622bc0249e2b848a0e430d9ec5aa4abf2f363c87f477965e9c1649f1e9d7347c"} Oct 30 00:37:25.446656 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:25.446630 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30713353-4nrqg" event={"ID":"da718472-0862-4f25-8043-057c7fb59d17","Type":"ContainerStarted","Data":"eb4457935fd63a50385a3d98250ab9a27ba5b408c89d31031ba16bae33e656ca"} Oct 30 00:37:25.446798 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:25.446674 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30713353-4nrqg" event={"ID":"da718472-0862-4f25-8043-057c7fb59d17","Type":"ContainerStarted","Data":"85a7080b45967429ee0cb0163543e830b69e364479f3d17cb6c4ce23e30f9276"} Oct 30 00:37:25.448742 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:25.448715 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d234d0-r994g" event={"ID":"f86ca908-fc4f-4c3a-a519-5d8f928576e3","Type":"ContainerStarted","Data":"44a2246230ed84302d11e8c343b4b29b4494068553caf1a1af7a62d41c6c4878"} Oct 30 00:37:25.448834 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:25.448769 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d234d0-r994g" event={"ID":"f86ca908-fc4f-4c3a-a519-5d8f928576e3","Type":"ContainerStarted","Data":"4f9d32cc3259909de9a045008480037498d23f558f7200dcc8d2d239511e7e4d"} Oct 30 00:37:25.450397 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:25.450376 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e58d8ab-cxfng" event={"ID":"0989dd93-7d85-4178-b9b0-48c87f0970a9","Type":"ContainerStarted","Data":"db5ac00aa9abc8e3c914120a37e2b09e0e2ea28a5fc179c467c3bd00ef9e7320"} Oct 30 00:37:25.483622 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:25.483574 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4e22fb-7kkcm" podStartSLOduration=1.483562666 podStartE2EDuration="1.483562666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:37:25.478720176 +0000 UTC m=+1960.684221356" watchObservedRunningTime="2026-10-30 00:37:25.483562666 +0000 UTC m=+1960.689063847" Oct 30 00:37:25.549175 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:25.549132 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d234d0-r994g" podStartSLOduration=2.549116078 podStartE2EDuration="2.549116078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:37:25.546640053 +0000 UTC m=+1960.752141234" watchObservedRunningTime="2026-10-30 00:37:25.549116078 +0000 UTC m=+1960.754617260" Oct 30 00:37:25.588375 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:25.588309 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30713353-4nrqg" podStartSLOduration=2.58829176 podStartE2EDuration="2.58829176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:37:25.580376944 +0000 UTC m=+1960.785878126" watchObservedRunningTime="2026-10-30 00:37:25.58829176 +0000 UTC m=+1960.793792942" Oct 30 00:37:26.460506 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:26.460470 5001 generic.go:356] "Generic (PLEG): container finished" podID="f86ca908-fc4f-4c3a-a519-5d8f928576e3" containerID="44a2246230ed84302d11e8c343b4b29b4494068553caf1a1af7a62d41c6c4878" exitCode=0 Oct 30 00:37:26.460965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:26.460553 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d234d0-r994g" event={"ID":"f86ca908-fc4f-4c3a-a519-5d8f928576e3","Type":"ContainerDied","Data":"44a2246230ed84302d11e8c343b4b29b4494068553caf1a1af7a62d41c6c4878"} Oct 30 00:37:26.462238 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:26.462215 5001 generic.go:356] "Generic (PLEG): container finished" podID="0989dd93-7d85-4178-b9b0-48c87f0970a9" containerID="db5ac00aa9abc8e3c914120a37e2b09e0e2ea28a5fc179c467c3bd00ef9e7320" exitCode=0 Oct 30 00:37:26.462386 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:26.462239 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e58d8ab-cxfng" event={"ID":"0989dd93-7d85-4178-b9b0-48c87f0970a9","Type":"ContainerDied","Data":"db5ac00aa9abc8e3c914120a37e2b09e0e2ea28a5fc179c467c3bd00ef9e7320"} Oct 30 00:37:26.464340 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:26.464320 5001 generic.go:356] "Generic (PLEG): container finished" podID="d51bbbf9-d605-49bb-ae09-426c7a5622c0" containerID="f08ac31a16523fd66c89e010a6253beabaa9f4feca8179c4677ac694bbb7dc5b" exitCode=0 Oct 30 00:37:26.464447 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:26.464397 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4e22fb-7kkcm" event={"ID":"d51bbbf9-d605-49bb-ae09-426c7a5622c0","Type":"ContainerDied","Data":"f08ac31a16523fd66c89e010a6253beabaa9f4feca8179c4677ac694bbb7dc5b"} Oct 30 00:37:26.466163 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:26.466146 5001 generic.go:356] "Generic (PLEG): container finished" podID="da718472-0862-4f25-8043-057c7fb59d17" containerID="eb4457935fd63a50385a3d98250ab9a27ba5b408c89d31031ba16bae33e656ca" exitCode=0 Oct 30 00:37:26.466265 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:26.466220 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30713353-4nrqg" event={"ID":"da718472-0862-4f25-8043-057c7fb59d17","Type":"ContainerDied","Data":"eb4457935fd63a50385a3d98250ab9a27ba5b408c89d31031ba16bae33e656ca"} Oct 30 00:37:26.646682 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:26.646655 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e58d8ab-cxfng" Oct 30 00:37:26.702422 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:26.702390 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0989dd93-7d85-4178-b9b0-48c87f0970a9-kube-api-access-crtxq\" (UniqueName: \"kubernetes.io/projected/0989dd93-7d85-4178-b9b0-48c87f0970a9-kube-api-access-crtxq\") pod \"0989dd93-7d85-4178-b9b0-48c87f0970a9\" (UID: \"0989dd93-7d85-4178-b9b0-48c87f0970a9\") " Oct 30 00:37:26.702627 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:26.702469 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0989dd93-7d85-4178-b9b0-48c87f0970a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0989dd93-7d85-4178-b9b0-48c87f0970a9-maas-api-openshift-ca\") pod \"0989dd93-7d85-4178-b9b0-48c87f0970a9\" (UID: \"0989dd93-7d85-4178-b9b0-48c87f0970a9\") " Oct 30 00:37:26.702627 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:26.702586 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0989dd93-7d85-4178-b9b0-48c87f0970a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0989dd93-7d85-4178-b9b0-48c87f0970a9-maas-api-xks-ca\") pod \"0989dd93-7d85-4178-b9b0-48c87f0970a9\" (UID: \"0989dd93-7d85-4178-b9b0-48c87f0970a9\") " Oct 30 00:37:26.702993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:26.702962 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0989dd93-7d85-4178-b9b0-48c87f0970a9-maas-api-openshift-ca" pod "0989dd93-7d85-4178-b9b0-48c87f0970a9" (UID: "0989dd93-7d85-4178-b9b0-48c87f0970a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:37:26.703318 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:26.703292 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0989dd93-7d85-4178-b9b0-48c87f0970a9-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:26.705320 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:26.705286 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0989dd93-7d85-4178-b9b0-48c87f0970a9-kube-api-access-crtxq" pod "0989dd93-7d85-4178-b9b0-48c87f0970a9" (UID: "0989dd93-7d85-4178-b9b0-48c87f0970a9"). InnerVolumeSpecName "kube-api-access-crtxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:26.706437 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:26.706397 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0989dd93-7d85-4178-b9b0-48c87f0970a9-maas-api-xks-ca" pod "0989dd93-7d85-4178-b9b0-48c87f0970a9" (UID: "0989dd93-7d85-4178-b9b0-48c87f0970a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:37:26.805216 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:26.805186 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0989dd93-7d85-4178-b9b0-48c87f0970a9-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:26.805216 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:26.805215 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crtxq\" (UniqueName: \"kubernetes.io/projected/0989dd93-7d85-4178-b9b0-48c87f0970a9-kube-api-access-crtxq\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:27.487613 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:27.487584 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e58d8ab-cxfng" Oct 30 00:37:27.489634 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:27.487805 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e58d8ab-cxfng" event={"ID":"0989dd93-7d85-4178-b9b0-48c87f0970a9","Type":"ContainerDied","Data":"2c1071a874229823c9fd3f76288c10b9e6691deff962f1e501d684febc63f759"} Oct 30 00:37:27.489634 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:27.489209 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c1071a874229823c9fd3f76288c10b9e6691deff962f1e501d684febc63f759" Oct 30 00:37:27.568380 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:27.568212 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" containerID="cri-o://14e4d2f1d6af626024fa624c224f87be29d12beb22b8cd096387bda75749abd0" gracePeriod=45 Oct 30 00:37:27.577119 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:27.576823 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" containerID="cri-o://ce73dcc9235be1427e6ac49d5fc1632c73882204fc02d2d8b1dc74b770657a4e" gracePeriod=45 Oct 30 00:37:27.577119 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:27.577008 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" containerID="cri-o://82894cd7709ff0c84e160ae993e5f6ae2bb893241ba0f3a422140306490c4a1b" gracePeriod=45 Oct 30 00:37:27.831106 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:27.830981 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": dial tcp 10.132.0.112:8000: connect: connection refused" Oct 30 00:37:27.834837 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:27.834811 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c1eeb6-sjr52"] Oct 30 00:37:27.836228 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:27.836203 5001 cpu_manager.go:414] "containerMap: removing container" podUID="0989dd93-7d85-4178-b9b0-48c87f0970a9" containerName="revoke-keys" Oct 30 00:37:27.836417 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:27.836401 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0989dd93-7d85-4178-b9b0-48c87f0970a9" containerName="revoke-keys" Oct 30 00:37:27.836936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:27.836902 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0989dd93-7d85-4178-b9b0-48c87f0970a9" containerName="revoke-keys" Oct 30 00:37:27.850203 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:27.849279 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c1eeb6-sjr52" Oct 30 00:37:27.853066 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:27.853032 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c1eeb6-sjr52"] Oct 30 00:37:27.930429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:27.930393 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bbp2\" (UniqueName: \"kubernetes.io/projected/67367a3a-0855-4843-b316-666d2654e6da-kube-api-access-8bbp2\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c1eeb6-sjr52\" (UID: \"67367a3a-0855-4843-b316-666d2654e6da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c1eeb6-sjr52" Oct 30 00:37:27.930589 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:27.930449 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67367a3a-0855-4843-b316-666d2654e6da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c1eeb6-sjr52\" (UID: \"67367a3a-0855-4843-b316-666d2654e6da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c1eeb6-sjr52" Oct 30 00:37:27.930589 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:27.930542 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67367a3a-0855-4843-b316-666d2654e6da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c1eeb6-sjr52\" (UID: \"67367a3a-0855-4843-b316-666d2654e6da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c1eeb6-sjr52" Oct 30 00:37:28.010718 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.010690 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4e22fb-7kkcm" Oct 30 00:37:28.034471 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.034442 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdf3c170-stq7z"] Oct 30 00:37:28.039389 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.039356 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bbp2\" (UniqueName: \"kubernetes.io/projected/67367a3a-0855-4843-b316-666d2654e6da-kube-api-access-8bbp2\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c1eeb6-sjr52\" (UID: \"67367a3a-0855-4843-b316-666d2654e6da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c1eeb6-sjr52" Oct 30 00:37:28.039526 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.039405 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67367a3a-0855-4843-b316-666d2654e6da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c1eeb6-sjr52\" (UID: \"67367a3a-0855-4843-b316-666d2654e6da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c1eeb6-sjr52" Oct 30 00:37:28.039526 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.039509 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67367a3a-0855-4843-b316-666d2654e6da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c1eeb6-sjr52\" (UID: \"67367a3a-0855-4843-b316-666d2654e6da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c1eeb6-sjr52" Oct 30 00:37:28.040609 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.040555 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67367a3a-0855-4843-b316-666d2654e6da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c1eeb6-sjr52\" (UID: \"67367a3a-0855-4843-b316-666d2654e6da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c1eeb6-sjr52" Oct 30 00:37:28.046449 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.045812 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdf3c170-stq7z"] Oct 30 00:37:28.046798 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.046778 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30713353-4nrqg" Oct 30 00:37:28.047873 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.047850 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67367a3a-0855-4843-b316-666d2654e6da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c1eeb6-sjr52\" (UID: \"67367a3a-0855-4843-b316-666d2654e6da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c1eeb6-sjr52" Oct 30 00:37:28.052014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.051992 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d234d0-r994g" Oct 30 00:37:28.075805 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.075774 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bbp2\" (UniqueName: \"kubernetes.io/projected/67367a3a-0855-4843-b316-666d2654e6da-kube-api-access-8bbp2\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c1eeb6-sjr52\" (UID: \"67367a3a-0855-4843-b316-666d2654e6da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c1eeb6-sjr52" Oct 30 00:37:28.140774 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.140739 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d51bbbf9-d605-49bb-ae09-426c7a5622c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d51bbbf9-d605-49bb-ae09-426c7a5622c0-maas-api-openshift-ca\") pod \"d51bbbf9-d605-49bb-ae09-426c7a5622c0\" (UID: \"d51bbbf9-d605-49bb-ae09-426c7a5622c0\") " Oct 30 00:37:28.140996 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.140857 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f86ca908-fc4f-4c3a-a519-5d8f928576e3-kube-api-access-czngw\" (UniqueName: \"kubernetes.io/projected/f86ca908-fc4f-4c3a-a519-5d8f928576e3-kube-api-access-czngw\") pod \"f86ca908-fc4f-4c3a-a519-5d8f928576e3\" (UID: \"f86ca908-fc4f-4c3a-a519-5d8f928576e3\") " Oct 30 00:37:28.141092 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.141022 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f86ca908-fc4f-4c3a-a519-5d8f928576e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f86ca908-fc4f-4c3a-a519-5d8f928576e3-maas-api-openshift-ca\") pod \"f86ca908-fc4f-4c3a-a519-5d8f928576e3\" (UID: \"f86ca908-fc4f-4c3a-a519-5d8f928576e3\") " Oct 30 00:37:28.141160 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.141103 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da718472-0862-4f25-8043-057c7fb59d17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da718472-0862-4f25-8043-057c7fb59d17-maas-api-xks-ca\") pod \"da718472-0862-4f25-8043-057c7fb59d17\" (UID: \"da718472-0862-4f25-8043-057c7fb59d17\") " Oct 30 00:37:28.141160 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.141134 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da718472-0862-4f25-8043-057c7fb59d17-kube-api-access-ntfjk\" (UniqueName: \"kubernetes.io/projected/da718472-0862-4f25-8043-057c7fb59d17-kube-api-access-ntfjk\") pod \"da718472-0862-4f25-8043-057c7fb59d17\" (UID: \"da718472-0862-4f25-8043-057c7fb59d17\") " Oct 30 00:37:28.141275 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.141169 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d51bbbf9-d605-49bb-ae09-426c7a5622c0-kube-api-access-wrw6b\" (UniqueName: \"kubernetes.io/projected/d51bbbf9-d605-49bb-ae09-426c7a5622c0-kube-api-access-wrw6b\") pod \"d51bbbf9-d605-49bb-ae09-426c7a5622c0\" (UID: \"d51bbbf9-d605-49bb-ae09-426c7a5622c0\") " Oct 30 00:37:28.141275 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.141198 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d51bbbf9-d605-49bb-ae09-426c7a5622c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d51bbbf9-d605-49bb-ae09-426c7a5622c0-maas-api-xks-ca\") pod \"d51bbbf9-d605-49bb-ae09-426c7a5622c0\" (UID: \"d51bbbf9-d605-49bb-ae09-426c7a5622c0\") " Oct 30 00:37:28.141275 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.141236 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f86ca908-fc4f-4c3a-a519-5d8f928576e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f86ca908-fc4f-4c3a-a519-5d8f928576e3-maas-api-xks-ca\") pod \"f86ca908-fc4f-4c3a-a519-5d8f928576e3\" (UID: \"f86ca908-fc4f-4c3a-a519-5d8f928576e3\") " Oct 30 00:37:28.141473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.141309 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da718472-0862-4f25-8043-057c7fb59d17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da718472-0862-4f25-8043-057c7fb59d17-maas-api-openshift-ca\") pod \"da718472-0862-4f25-8043-057c7fb59d17\" (UID: \"da718472-0862-4f25-8043-057c7fb59d17\") " Oct 30 00:37:28.141473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.141355 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d51bbbf9-d605-49bb-ae09-426c7a5622c0-maas-api-openshift-ca" pod "d51bbbf9-d605-49bb-ae09-426c7a5622c0" (UID: "d51bbbf9-d605-49bb-ae09-426c7a5622c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:37:28.142133 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.142107 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d51bbbf9-d605-49bb-ae09-426c7a5622c0-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:28.142308 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.142291 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da718472-0862-4f25-8043-057c7fb59d17-maas-api-openshift-ca" pod "da718472-0862-4f25-8043-057c7fb59d17" (UID: "da718472-0862-4f25-8043-057c7fb59d17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:37:28.142855 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.142816 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f86ca908-fc4f-4c3a-a519-5d8f928576e3-maas-api-openshift-ca" pod "f86ca908-fc4f-4c3a-a519-5d8f928576e3" (UID: "f86ca908-fc4f-4c3a-a519-5d8f928576e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:37:28.150276 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.150246 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da718472-0862-4f25-8043-057c7fb59d17-maas-api-xks-ca" pod "da718472-0862-4f25-8043-057c7fb59d17" (UID: "da718472-0862-4f25-8043-057c7fb59d17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:37:28.150383 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.150269 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f86ca908-fc4f-4c3a-a519-5d8f928576e3-kube-api-access-czngw" pod "f86ca908-fc4f-4c3a-a519-5d8f928576e3" (UID: "f86ca908-fc4f-4c3a-a519-5d8f928576e3"). InnerVolumeSpecName "kube-api-access-czngw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:28.150383 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.150313 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da718472-0862-4f25-8043-057c7fb59d17-kube-api-access-ntfjk" pod "da718472-0862-4f25-8043-057c7fb59d17" (UID: "da718472-0862-4f25-8043-057c7fb59d17"). InnerVolumeSpecName "kube-api-access-ntfjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:28.151655 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.151633 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d51bbbf9-d605-49bb-ae09-426c7a5622c0-maas-api-xks-ca" pod "d51bbbf9-d605-49bb-ae09-426c7a5622c0" (UID: "d51bbbf9-d605-49bb-ae09-426c7a5622c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:37:28.152655 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.152631 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f86ca908-fc4f-4c3a-a519-5d8f928576e3-maas-api-xks-ca" pod "f86ca908-fc4f-4c3a-a519-5d8f928576e3" (UID: "f86ca908-fc4f-4c3a-a519-5d8f928576e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:37:28.153058 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.153036 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d51bbbf9-d605-49bb-ae09-426c7a5622c0-kube-api-access-wrw6b" pod "d51bbbf9-d605-49bb-ae09-426c7a5622c0" (UID: "d51bbbf9-d605-49bb-ae09-426c7a5622c0"). InnerVolumeSpecName "kube-api-access-wrw6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:28.162566 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.162518 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:37:28.242953 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.242878 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde1d5db-cea4-4020-b843-2321a1e346c4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dde1d5db-cea4-4020-b843-2321a1e346c4-tmp-dir\") pod \"dde1d5db-cea4-4020-b843-2321a1e346c4\" (UID: \"dde1d5db-cea4-4020-b843-2321a1e346c4\") " Oct 30 00:37:28.243171 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.243128 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde1d5db-cea4-4020-b843-2321a1e346c4-home\" (UniqueName: \"kubernetes.io/empty-dir/dde1d5db-cea4-4020-b843-2321a1e346c4-home\") pod \"dde1d5db-cea4-4020-b843-2321a1e346c4\" (UID: \"dde1d5db-cea4-4020-b843-2321a1e346c4\") " Oct 30 00:37:28.243255 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.243194 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dde1d5db-cea4-4020-b843-2321a1e346c4-tls-certs\" (UniqueName: \"kubernetes.io/secret/dde1d5db-cea4-4020-b843-2321a1e346c4-tls-certs\") pod \"dde1d5db-cea4-4020-b843-2321a1e346c4\" (UID: \"dde1d5db-cea4-4020-b843-2321a1e346c4\") " Oct 30 00:37:28.243255 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.243214 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde1d5db-cea4-4020-b843-2321a1e346c4-tmp-dir" pod "dde1d5db-cea4-4020-b843-2321a1e346c4" (UID: "dde1d5db-cea4-4020-b843-2321a1e346c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:28.243392 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.243298 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde1d5db-cea4-4020-b843-2321a1e346c4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dde1d5db-cea4-4020-b843-2321a1e346c4-dshm\") pod \"dde1d5db-cea4-4020-b843-2321a1e346c4\" (UID: \"dde1d5db-cea4-4020-b843-2321a1e346c4\") " Oct 30 00:37:28.243451 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.243411 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde1d5db-cea4-4020-b843-2321a1e346c4-home" pod "dde1d5db-cea4-4020-b843-2321a1e346c4" (UID: "dde1d5db-cea4-4020-b843-2321a1e346c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:28.243508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.243455 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde1d5db-cea4-4020-b843-2321a1e346c4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dde1d5db-cea4-4020-b843-2321a1e346c4-model-cache\") pod \"dde1d5db-cea4-4020-b843-2321a1e346c4\" (UID: \"dde1d5db-cea4-4020-b843-2321a1e346c4\") " Oct 30 00:37:28.243570 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.243523 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dde1d5db-cea4-4020-b843-2321a1e346c4-kube-api-access-2ps2t\" (UniqueName: \"kubernetes.io/projected/dde1d5db-cea4-4020-b843-2321a1e346c4-kube-api-access-2ps2t\") pod \"dde1d5db-cea4-4020-b843-2321a1e346c4\" (UID: \"dde1d5db-cea4-4020-b843-2321a1e346c4\") " Oct 30 00:37:28.243702 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.243677 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde1d5db-cea4-4020-b843-2321a1e346c4-model-cache" pod "dde1d5db-cea4-4020-b843-2321a1e346c4" (UID: "dde1d5db-cea4-4020-b843-2321a1e346c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:28.244426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.244402 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da718472-0862-4f25-8043-057c7fb59d17-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:28.244540 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.244432 5001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dde1d5db-cea4-4020-b843-2321a1e346c4-model-cache\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:28.244540 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.244449 5001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dde1d5db-cea4-4020-b843-2321a1e346c4-tmp-dir\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:28.244540 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.244461 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czngw\" (UniqueName: \"kubernetes.io/projected/f86ca908-fc4f-4c3a-a519-5d8f928576e3-kube-api-access-czngw\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:28.244540 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.244476 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f86ca908-fc4f-4c3a-a519-5d8f928576e3-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:28.244540 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.244488 5001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dde1d5db-cea4-4020-b843-2321a1e346c4-home\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:28.244540 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.244502 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da718472-0862-4f25-8043-057c7fb59d17-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:28.244540 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.244515 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntfjk\" (UniqueName: \"kubernetes.io/projected/da718472-0862-4f25-8043-057c7fb59d17-kube-api-access-ntfjk\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:28.244540 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.244528 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrw6b\" (UniqueName: \"kubernetes.io/projected/d51bbbf9-d605-49bb-ae09-426c7a5622c0-kube-api-access-wrw6b\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:28.244540 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.244542 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d51bbbf9-d605-49bb-ae09-426c7a5622c0-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:28.245090 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.244555 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f86ca908-fc4f-4c3a-a519-5d8f928576e3-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:28.245983 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.245952 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde1d5db-cea4-4020-b843-2321a1e346c4-dshm" pod "dde1d5db-cea4-4020-b843-2321a1e346c4" (UID: "dde1d5db-cea4-4020-b843-2321a1e346c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:28.246095 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.246013 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dde1d5db-cea4-4020-b843-2321a1e346c4-tls-certs" pod "dde1d5db-cea4-4020-b843-2321a1e346c4" (UID: "dde1d5db-cea4-4020-b843-2321a1e346c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:37:28.246095 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.246046 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dde1d5db-cea4-4020-b843-2321a1e346c4-kube-api-access-2ps2t" pod "dde1d5db-cea4-4020-b843-2321a1e346c4" (UID: "dde1d5db-cea4-4020-b843-2321a1e346c4"). InnerVolumeSpecName "kube-api-access-2ps2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:28.309536 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.309500 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c1eeb6-sjr52" Oct 30 00:37:28.346840 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.346802 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ps2t\" (UniqueName: \"kubernetes.io/projected/dde1d5db-cea4-4020-b843-2321a1e346c4-kube-api-access-2ps2t\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:28.346994 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.346846 5001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dde1d5db-cea4-4020-b843-2321a1e346c4-tls-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:28.346994 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.346860 5001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dde1d5db-cea4-4020-b843-2321a1e346c4-dshm\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:28.348167 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.348060 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": dial tcp 10.132.0.100:8000: connect: connection refused" Oct 30 00:37:28.477474 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.477447 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:37:28.501462 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.501429 5001 generic.go:356] "Generic (PLEG): container finished" podID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerID="14e4d2f1d6af626024fa624c224f87be29d12beb22b8cd096387bda75749abd0" exitCode=0 Oct 30 00:37:28.501883 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.501538 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" event={"ID":"2f0cfc05-3778-4a07-b822-cffc4d94ddb6","Type":"ContainerDied","Data":"14e4d2f1d6af626024fa624c224f87be29d12beb22b8cd096387bda75749abd0"} Oct 30 00:37:28.510055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.503756 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d234d0-r994g" event={"ID":"f86ca908-fc4f-4c3a-a519-5d8f928576e3","Type":"ContainerDied","Data":"4f9d32cc3259909de9a045008480037498d23f558f7200dcc8d2d239511e7e4d"} Oct 30 00:37:28.510055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.503808 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f9d32cc3259909de9a045008480037498d23f558f7200dcc8d2d239511e7e4d" Oct 30 00:37:28.510055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.503871 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d234d0-r994g" Oct 30 00:37:28.513200 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.513160 5001 generic.go:356] "Generic (PLEG): container finished" podID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerID="ce73dcc9235be1427e6ac49d5fc1632c73882204fc02d2d8b1dc74b770657a4e" exitCode=0 Oct 30 00:37:28.513350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.513298 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" event={"ID":"dde1d5db-cea4-4020-b843-2321a1e346c4","Type":"ContainerDied","Data":"ce73dcc9235be1427e6ac49d5fc1632c73882204fc02d2d8b1dc74b770657a4e"} Oct 30 00:37:28.513350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.513333 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" event={"ID":"dde1d5db-cea4-4020-b843-2321a1e346c4","Type":"ContainerDied","Data":"324e3a5f3b25ba7249d7d09063fe90ec98fdc4c856128548efed31849a0d5c8f"} Oct 30 00:37:28.513481 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.513355 5001 scope.go:122] "RemoveContainer" containerID="ce73dcc9235be1427e6ac49d5fc1632c73882204fc02d2d8b1dc74b770657a4e" Oct 30 00:37:28.513550 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.513529 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" Oct 30 00:37:28.551250 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.550973 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-home\" (UniqueName: \"kubernetes.io/empty-dir/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-home\") pod \"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f\" (UID: \"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f\") " Oct 30 00:37:28.551250 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.551145 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-dshm\") pod \"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f\" (UID: \"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f\") " Oct 30 00:37:28.551394 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.551247 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-home" pod "b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" (UID: "b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:28.551394 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.551282 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-tmp-dir\") pod \"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f\" (UID: \"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f\") " Oct 30 00:37:28.551394 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.551354 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-model-cache\") pod \"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f\" (UID: \"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f\") " Oct 30 00:37:28.552008 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.551828 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-tmp-dir" pod "b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" (UID: "b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:28.552120 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.552075 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4e22fb-7kkcm" event={"ID":"d51bbbf9-d605-49bb-ae09-426c7a5622c0","Type":"ContainerDied","Data":"622bc0249e2b848a0e430d9ec5aa4abf2f363c87f477965e9c1649f1e9d7347c"} Oct 30 00:37:28.552120 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.552104 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4e22fb-7kkcm" Oct 30 00:37:28.552259 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.552169 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="622bc0249e2b848a0e430d9ec5aa4abf2f363c87f477965e9c1649f1e9d7347c" Oct 30 00:37:28.554117 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.551383 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-tls-certs\" (UniqueName: \"kubernetes.io/secret/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-tls-certs\") pod \"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f\" (UID: \"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f\") " Oct 30 00:37:28.554252 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.554160 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-kube-api-access-dmphn\" (UniqueName: \"kubernetes.io/projected/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-kube-api-access-dmphn\") pod \"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f\" (UID: \"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f\") " Oct 30 00:37:28.555172 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.555121 5001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-home\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:28.555172 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.555148 5001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-tmp-dir\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:28.555336 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.555167 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-model-cache" pod "b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" (UID: "b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:28.557591 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.557568 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-kube-api-access-dmphn" pod "b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" (UID: "b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f"). InnerVolumeSpecName "kube-api-access-dmphn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:28.558002 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.557978 5001 generic.go:356] "Generic (PLEG): container finished" podID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerID="82894cd7709ff0c84e160ae993e5f6ae2bb893241ba0f3a422140306490c4a1b" exitCode=0 Oct 30 00:37:28.558146 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.558122 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" event={"ID":"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f","Type":"ContainerDied","Data":"82894cd7709ff0c84e160ae993e5f6ae2bb893241ba0f3a422140306490c4a1b"} Oct 30 00:37:28.558214 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.558165 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" event={"ID":"b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f","Type":"ContainerDied","Data":"70e158d2104c64a9bbd70419391851dd9effb43600f3838f10dabf89470b4faf"} Oct 30 00:37:28.558279 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.558259 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh" Oct 30 00:37:28.561313 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.561289 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-dshm" pod "b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" (UID: "b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:28.561897 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.561870 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-tls-certs" pod "b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" (UID: "b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:37:28.562122 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.562084 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30713353-4nrqg" event={"ID":"da718472-0862-4f25-8043-057c7fb59d17","Type":"ContainerDied","Data":"85a7080b45967429ee0cb0163543e830b69e364479f3d17cb6c4ce23e30f9276"} Oct 30 00:37:28.562191 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.562143 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85a7080b45967429ee0cb0163543e830b69e364479f3d17cb6c4ce23e30f9276" Oct 30 00:37:28.562191 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.562110 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30713353-4nrqg" Oct 30 00:37:28.566151 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.566004 5001 scope.go:122] "RemoveContainer" containerID="ce73dcc9235be1427e6ac49d5fc1632c73882204fc02d2d8b1dc74b770657a4e" Oct 30 00:37:28.566355 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.566266 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:37:28.566849 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:37:28.566707 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce73dcc9235be1427e6ac49d5fc1632c73882204fc02d2d8b1dc74b770657a4e\": container with ID starting with ce73dcc9235be1427e6ac49d5fc1632c73882204fc02d2d8b1dc74b770657a4e not found: ID does not exist" containerID="ce73dcc9235be1427e6ac49d5fc1632c73882204fc02d2d8b1dc74b770657a4e" Oct 30 00:37:28.566849 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.566741 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce73dcc9235be1427e6ac49d5fc1632c73882204fc02d2d8b1dc74b770657a4e"} err="failed to get container status \"ce73dcc9235be1427e6ac49d5fc1632c73882204fc02d2d8b1dc74b770657a4e\": rpc error: code = NotFound desc = could not find container \"ce73dcc9235be1427e6ac49d5fc1632c73882204fc02d2d8b1dc74b770657a4e\": container with ID starting with ce73dcc9235be1427e6ac49d5fc1632c73882204fc02d2d8b1dc74b770657a4e not found: ID does not exist" Oct 30 00:37:28.566849 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.566763 5001 scope.go:122] "RemoveContainer" containerID="82894cd7709ff0c84e160ae993e5f6ae2bb893241ba0f3a422140306490c4a1b" Oct 30 00:37:28.581843 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.581817 5001 scope.go:122] "RemoveContainer" containerID="82894cd7709ff0c84e160ae993e5f6ae2bb893241ba0f3a422140306490c4a1b" Oct 30 00:37:28.582222 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:37:28.582196 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82894cd7709ff0c84e160ae993e5f6ae2bb893241ba0f3a422140306490c4a1b\": container with ID starting with 82894cd7709ff0c84e160ae993e5f6ae2bb893241ba0f3a422140306490c4a1b not found: ID does not exist" containerID="82894cd7709ff0c84e160ae993e5f6ae2bb893241ba0f3a422140306490c4a1b" Oct 30 00:37:28.582335 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.582228 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82894cd7709ff0c84e160ae993e5f6ae2bb893241ba0f3a422140306490c4a1b"} err="failed to get container status \"82894cd7709ff0c84e160ae993e5f6ae2bb893241ba0f3a422140306490c4a1b\": rpc error: code = NotFound desc = could not find container \"82894cd7709ff0c84e160ae993e5f6ae2bb893241ba0f3a422140306490c4a1b\": container with ID starting with 82894cd7709ff0c84e160ae993e5f6ae2bb893241ba0f3a422140306490c4a1b not found: ID does not exist" Oct 30 00:37:28.590250 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.590218 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq"] Oct 30 00:37:28.596056 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.596034 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq"] Oct 30 00:37:28.650411 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:37:28.650378 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67367a3a_0855_4843_b316_666d2654e6da.slice/crio-adc8e6f115630de9eecf47adf89bfea192f0b16ffc55e95f43b79ef77ee9e46b WatchSource:0}: Error finding container adc8e6f115630de9eecf47adf89bfea192f0b16ffc55e95f43b79ef77ee9e46b: Status 404 returned error can't find the container with id adc8e6f115630de9eecf47adf89bfea192f0b16ffc55e95f43b79ef77ee9e46b Oct 30 00:37:28.650785 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.650762 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c1eeb6-sjr52"] Oct 30 00:37:28.656111 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.656090 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-kube-api-access-ndrvd\" (UniqueName: \"kubernetes.io/projected/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-kube-api-access-ndrvd\") pod \"2f0cfc05-3778-4a07-b822-cffc4d94ddb6\" (UID: \"2f0cfc05-3778-4a07-b822-cffc4d94ddb6\") " Oct 30 00:37:28.656210 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.656178 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-tls-certs\" (UniqueName: \"kubernetes.io/secret/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-tls-certs\") pod \"2f0cfc05-3778-4a07-b822-cffc4d94ddb6\" (UID: \"2f0cfc05-3778-4a07-b822-cffc4d94ddb6\") " Oct 30 00:37:28.656274 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.656264 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-dshm\") pod \"2f0cfc05-3778-4a07-b822-cffc4d94ddb6\" (UID: \"2f0cfc05-3778-4a07-b822-cffc4d94ddb6\") " Oct 30 00:37:28.656341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.656291 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-model-cache\") pod \"2f0cfc05-3778-4a07-b822-cffc4d94ddb6\" (UID: \"2f0cfc05-3778-4a07-b822-cffc4d94ddb6\") " Oct 30 00:37:28.656341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.656321 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-tmp-dir\") pod \"2f0cfc05-3778-4a07-b822-cffc4d94ddb6\" (UID: \"2f0cfc05-3778-4a07-b822-cffc4d94ddb6\") " Oct 30 00:37:28.656477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.656461 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-home\" (UniqueName: \"kubernetes.io/empty-dir/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-home\") pod \"2f0cfc05-3778-4a07-b822-cffc4d94ddb6\" (UID: \"2f0cfc05-3778-4a07-b822-cffc4d94ddb6\") " Oct 30 00:37:28.656640 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.656595 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-model-cache" pod "2f0cfc05-3778-4a07-b822-cffc4d94ddb6" (UID: "2f0cfc05-3778-4a07-b822-cffc4d94ddb6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:28.656707 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.656637 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-tmp-dir" pod "2f0cfc05-3778-4a07-b822-cffc4d94ddb6" (UID: "2f0cfc05-3778-4a07-b822-cffc4d94ddb6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:28.656758 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.656739 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-home" pod "2f0cfc05-3778-4a07-b822-cffc4d94ddb6" (UID: "2f0cfc05-3778-4a07-b822-cffc4d94ddb6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:28.656999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.656982 5001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-model-cache\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:28.657131 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.657112 5001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-tmp-dir\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:28.657131 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.657132 5001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-model-cache\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:28.657349 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.657141 5001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-tls-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:28.657349 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.657150 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmphn\" (UniqueName: \"kubernetes.io/projected/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-kube-api-access-dmphn\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:28.657349 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.657161 5001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-home\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:28.657349 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.657173 5001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f-dshm\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:28.658510 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.658484 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-dshm" pod "2f0cfc05-3778-4a07-b822-cffc4d94ddb6" (UID: "2f0cfc05-3778-4a07-b822-cffc4d94ddb6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:28.658800 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.658780 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-tls-certs" pod "2f0cfc05-3778-4a07-b822-cffc4d94ddb6" (UID: "2f0cfc05-3778-4a07-b822-cffc4d94ddb6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:37:28.658872 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.658814 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-kube-api-access-ndrvd" pod "2f0cfc05-3778-4a07-b822-cffc4d94ddb6" (UID: "2f0cfc05-3778-4a07-b822-cffc4d94ddb6"). InnerVolumeSpecName "kube-api-access-ndrvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:28.704252 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.704217 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a7ac6-75b958fbc4-dqgk7" podUID="30bf357c-11ff-4fc6-bcfe-c776a3370b9f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:37:28.704514 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.704455 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a7ac6-68f8c58595-vb58b" podUID="776a2d19-ec0d-4868-ac0c-e22a7c4b6de0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:37:28.758601 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.758564 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndrvd\" (UniqueName: \"kubernetes.io/projected/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-kube-api-access-ndrvd\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:28.758601 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.758598 5001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-tls-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:28.758766 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.758611 5001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f0cfc05-3778-4a07-b822-cffc4d94ddb6-dshm\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:28.902903 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.902871 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh"] Oct 30 00:37:28.908105 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.908074 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-7a7ac6/distinct-w1-7a7ac6-kserve-66b96fff49-gbcqh"] Oct 30 00:37:28.920484 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.920461 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" path="/var/lib/kubelet/pods/b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f/volumes" Oct 30 00:37:28.920990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.920974 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" path="/var/lib/kubelet/pods/dde1d5db-cea4-4020-b843-2321a1e346c4/volumes" Oct 30 00:37:28.921453 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:28.921438 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2f023ad-b552-4c5a-89bf-aac6755064f1" path="/var/lib/kubelet/pods/f2f023ad-b552-4c5a-89bf-aac6755064f1/volumes" Oct 30 00:37:29.082076 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:29.081994 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8e7fdd9-tlbpv"] Oct 30 00:37:29.087124 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:29.087098 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8e7fdd9-tlbpv"] Oct 30 00:37:29.133064 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:29.133016 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7a7ac6/premium-w1-7a7ac6-kserve-5f667bbb5d-tzmmq" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:37:29.573757 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:29.573718 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c1eeb6-sjr52" event={"ID":"67367a3a-0855-4843-b316-666d2654e6da","Type":"ContainerStarted","Data":"4549a67c009d66b8518d952962c573a3ae408c82deb1e9922eb4cdd784a64fe8"} Oct 30 00:37:29.574184 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:29.573779 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c1eeb6-sjr52" event={"ID":"67367a3a-0855-4843-b316-666d2654e6da","Type":"ContainerStarted","Data":"adc8e6f115630de9eecf47adf89bfea192f0b16ffc55e95f43b79ef77ee9e46b"} Oct 30 00:37:29.575443 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:29.575419 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" Oct 30 00:37:29.575590 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:29.575454 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc" event={"ID":"2f0cfc05-3778-4a07-b822-cffc4d94ddb6","Type":"ContainerDied","Data":"ceee601082f5232937498db37807613a689f175d34b2c5b7e5ea19ca360db5c8"} Oct 30 00:37:29.575590 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:29.575498 5001 scope.go:122] "RemoveContainer" containerID="14e4d2f1d6af626024fa624c224f87be29d12beb22b8cd096387bda75749abd0" Oct 30 00:37:29.619115 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:29.619069 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c1eeb6-sjr52" podStartSLOduration=2.61905664 podStartE2EDuration="2.61905664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:37:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:37:29.612775338 +0000 UTC m=+1964.818276521" watchObservedRunningTime="2026-10-30 00:37:29.61905664 +0000 UTC m=+1964.824557839" Oct 30 00:37:29.647754 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:29.647727 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc"] Oct 30 00:37:29.653648 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:29.653626 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-7a7ac6/distinct-2-w1-7a7ac6-kserve-7b567498b6-q5gnc"] Oct 30 00:37:30.588344 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:30.588309 5001 generic.go:356] "Generic (PLEG): container finished" podID="67367a3a-0855-4843-b316-666d2654e6da" containerID="4549a67c009d66b8518d952962c573a3ae408c82deb1e9922eb4cdd784a64fe8" exitCode=0 Oct 30 00:37:30.588785 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:30.588391 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c1eeb6-sjr52" event={"ID":"67367a3a-0855-4843-b316-666d2654e6da","Type":"ContainerDied","Data":"4549a67c009d66b8518d952962c573a3ae408c82deb1e9922eb4cdd784a64fe8"} Oct 30 00:37:30.939397 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:30.930450 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1832bfed-6454-4368-bcd5-223c048fcc36" path="/var/lib/kubelet/pods/1832bfed-6454-4368-bcd5-223c048fcc36/volumes" Oct 30 00:37:30.939397 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:30.931213 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" path="/var/lib/kubelet/pods/2f0cfc05-3778-4a07-b822-cffc4d94ddb6/volumes" Oct 30 00:37:31.762937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:31.762898 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c1eeb6-sjr52" Oct 30 00:37:31.816967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:31.816937 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67367a3a-0855-4843-b316-666d2654e6da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67367a3a-0855-4843-b316-666d2654e6da-maas-api-xks-ca\") pod \"67367a3a-0855-4843-b316-666d2654e6da\" (UID: \"67367a3a-0855-4843-b316-666d2654e6da\") " Oct 30 00:37:31.817116 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:31.816981 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67367a3a-0855-4843-b316-666d2654e6da-kube-api-access-8bbp2\" (UniqueName: \"kubernetes.io/projected/67367a3a-0855-4843-b316-666d2654e6da-kube-api-access-8bbp2\") pod \"67367a3a-0855-4843-b316-666d2654e6da\" (UID: \"67367a3a-0855-4843-b316-666d2654e6da\") " Oct 30 00:37:31.817116 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:31.817057 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67367a3a-0855-4843-b316-666d2654e6da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67367a3a-0855-4843-b316-666d2654e6da-maas-api-openshift-ca\") pod \"67367a3a-0855-4843-b316-666d2654e6da\" (UID: \"67367a3a-0855-4843-b316-666d2654e6da\") " Oct 30 00:37:31.817446 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:31.817418 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67367a3a-0855-4843-b316-666d2654e6da-maas-api-openshift-ca" pod "67367a3a-0855-4843-b316-666d2654e6da" (UID: "67367a3a-0855-4843-b316-666d2654e6da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:37:31.817699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:31.817679 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67367a3a-0855-4843-b316-666d2654e6da-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:31.819186 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:31.819159 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67367a3a-0855-4843-b316-666d2654e6da-maas-api-xks-ca" pod "67367a3a-0855-4843-b316-666d2654e6da" (UID: "67367a3a-0855-4843-b316-666d2654e6da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:37:31.819620 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:31.819603 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67367a3a-0855-4843-b316-666d2654e6da-kube-api-access-8bbp2" pod "67367a3a-0855-4843-b316-666d2654e6da" (UID: "67367a3a-0855-4843-b316-666d2654e6da"). InnerVolumeSpecName "kube-api-access-8bbp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:31.919392 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:31.919320 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67367a3a-0855-4843-b316-666d2654e6da-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:31.919392 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:31.919349 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bbp2\" (UniqueName: \"kubernetes.io/projected/67367a3a-0855-4843-b316-666d2654e6da-kube-api-access-8bbp2\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:32.611925 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:32.611890 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c1eeb6-sjr52" Oct 30 00:37:32.612077 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:32.611954 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c1eeb6-sjr52" event={"ID":"67367a3a-0855-4843-b316-666d2654e6da","Type":"ContainerDied","Data":"adc8e6f115630de9eecf47adf89bfea192f0b16ffc55e95f43b79ef77ee9e46b"} Oct 30 00:37:32.612077 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:32.612000 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adc8e6f115630de9eecf47adf89bfea192f0b16ffc55e95f43b79ef77ee9e46b" Oct 30 00:37:38.032319 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:38.032281 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f3e21ed-vcpck"] Oct 30 00:37:38.041377 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:38.041343 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f3e21ed-vcpck"] Oct 30 00:37:38.231475 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:38.231439 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c1eeb6-7d97d86c79-zmp4b"] Oct 30 00:37:38.770935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:38.763450 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a7ac6-68f8c58595-vb58b" podUID="776a2d19-ec0d-4868-ac0c-e22a7c4b6de0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:37:38.802070 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:38.802025 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a7ac6-75b958fbc4-dqgk7" podUID="30bf357c-11ff-4fc6-bcfe-c776a3370b9f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:37:39.026608 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:39.026492 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c356c3ec-def9-47ab-ad8c-88620887acac" path="/var/lib/kubelet/pods/c356c3ec-def9-47ab-ad8c-88620887acac/volumes" Oct 30 00:37:39.039664 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:39.039624 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d034487-g9qjg"] Oct 30 00:37:39.044795 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:39.044768 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d034487-g9qjg"] Oct 30 00:37:39.768971 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:39.767965 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c1eeb6-sjr52"] Oct 30 00:37:39.778200 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:39.777678 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c1eeb6-sjr52"] Oct 30 00:37:41.020934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:41.020748 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="253eb22e-11d4-47c6-be47-40f024aff63d" path="/var/lib/kubelet/pods/253eb22e-11d4-47c6-be47-40f024aff63d/volumes" Oct 30 00:37:41.028933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:41.024024 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67367a3a-0855-4843-b316-666d2654e6da" path="/var/lib/kubelet/pods/67367a3a-0855-4843-b316-666d2654e6da/volumes" Oct 30 00:37:43.232524 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:43.232481 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c1eeb6-7d97d86c79-zmp4b" podUID="2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c" containerName="payload-pre-processing" containerID="cri-o://83e85ac636e6cbf91083e1187a5c2965a1a6265f3c27a4ecde281ec1daec8ecb" gracePeriod=25 Oct 30 00:37:43.647615 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:43.647589 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a7ac6-75b958fbc4-dqgk7" Oct 30 00:37:43.651047 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:43.651029 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a7ac6-68f8c58595-vb58b" Oct 30 00:37:43.747685 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:43.747643 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/776a2d19-ec0d-4868-ac0c-e22a7c4b6de0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/776a2d19-ec0d-4868-ac0c-e22a7c4b6de0-plugins-config-volume\") pod \"776a2d19-ec0d-4868-ac0c-e22a7c4b6de0\" (UID: \"776a2d19-ec0d-4868-ac0c-e22a7c4b6de0\") " Oct 30 00:37:43.747863 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:43.747813 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/776a2d19-ec0d-4868-ac0c-e22a7c4b6de0-kube-api-access-5smbl\" (UniqueName: \"kubernetes.io/projected/776a2d19-ec0d-4868-ac0c-e22a7c4b6de0-kube-api-access-5smbl\") pod \"776a2d19-ec0d-4868-ac0c-e22a7c4b6de0\" (UID: \"776a2d19-ec0d-4868-ac0c-e22a7c4b6de0\") " Oct 30 00:37:43.747965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:43.747944 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30bf357c-11ff-4fc6-bcfe-c776a3370b9f-kube-api-access-8lcqn\" (UniqueName: \"kubernetes.io/projected/30bf357c-11ff-4fc6-bcfe-c776a3370b9f-kube-api-access-8lcqn\") pod \"30bf357c-11ff-4fc6-bcfe-c776a3370b9f\" (UID: \"30bf357c-11ff-4fc6-bcfe-c776a3370b9f\") " Oct 30 00:37:43.748046 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:43.747996 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30bf357c-11ff-4fc6-bcfe-c776a3370b9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30bf357c-11ff-4fc6-bcfe-c776a3370b9f-plugins-config-volume\") pod \"30bf357c-11ff-4fc6-bcfe-c776a3370b9f\" (UID: \"30bf357c-11ff-4fc6-bcfe-c776a3370b9f\") " Oct 30 00:37:43.748258 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:43.748228 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/776a2d19-ec0d-4868-ac0c-e22a7c4b6de0-plugins-config-volume" pod "776a2d19-ec0d-4868-ac0c-e22a7c4b6de0" (UID: "776a2d19-ec0d-4868-ac0c-e22a7c4b6de0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:37:43.749580 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:43.748508 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30bf357c-11ff-4fc6-bcfe-c776a3370b9f-plugins-config-volume" pod "30bf357c-11ff-4fc6-bcfe-c776a3370b9f" (UID: "30bf357c-11ff-4fc6-bcfe-c776a3370b9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:37:43.752068 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:43.752036 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776a2d19-ec0d-4868-ac0c-e22a7c4b6de0-kube-api-access-5smbl" pod "776a2d19-ec0d-4868-ac0c-e22a7c4b6de0" (UID: "776a2d19-ec0d-4868-ac0c-e22a7c4b6de0"). InnerVolumeSpecName "kube-api-access-5smbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:43.752318 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:43.751998 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30bf357c-11ff-4fc6-bcfe-c776a3370b9f-kube-api-access-8lcqn" pod "30bf357c-11ff-4fc6-bcfe-c776a3370b9f" (UID: "30bf357c-11ff-4fc6-bcfe-c776a3370b9f"). InnerVolumeSpecName "kube-api-access-8lcqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:43.851141 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:43.851057 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30bf357c-11ff-4fc6-bcfe-c776a3370b9f-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:43.851141 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:43.851091 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/776a2d19-ec0d-4868-ac0c-e22a7c4b6de0-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:43.851141 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:43.851101 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5smbl\" (UniqueName: \"kubernetes.io/projected/776a2d19-ec0d-4868-ac0c-e22a7c4b6de0-kube-api-access-5smbl\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:43.851141 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:43.851110 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lcqn\" (UniqueName: \"kubernetes.io/projected/30bf357c-11ff-4fc6-bcfe-c776a3370b9f-kube-api-access-8lcqn\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:43.879940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:43.879895 5001 generic.go:356] "Generic (PLEG): container finished" podID="776a2d19-ec0d-4868-ac0c-e22a7c4b6de0" containerID="14c8a467de7b1d26a1a5858f03e940d85a951858ef045595a7184ff2c77a57d8" exitCode=0 Oct 30 00:37:43.880153 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:43.879992 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a7ac6-68f8c58595-vb58b" event={"ID":"776a2d19-ec0d-4868-ac0c-e22a7c4b6de0","Type":"ContainerDied","Data":"14c8a467de7b1d26a1a5858f03e940d85a951858ef045595a7184ff2c77a57d8"} Oct 30 00:37:43.880153 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:43.880093 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a7ac6-68f8c58595-vb58b" event={"ID":"776a2d19-ec0d-4868-ac0c-e22a7c4b6de0","Type":"ContainerDied","Data":"7a5b4d0dc7623448759816b9eacb0dd080eb06d4ce946a9dad9b7a1c01279a0e"} Oct 30 00:37:43.880153 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:43.880011 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a7ac6-68f8c58595-vb58b" Oct 30 00:37:43.880153 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:43.880126 5001 scope.go:122] "RemoveContainer" containerID="14c8a467de7b1d26a1a5858f03e940d85a951858ef045595a7184ff2c77a57d8" Oct 30 00:37:43.884330 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:43.884306 5001 generic.go:356] "Generic (PLEG): container finished" podID="30bf357c-11ff-4fc6-bcfe-c776a3370b9f" containerID="f2583b68cc5f20b95025592720f04fabb73f3a6d76b0377ba09a08df4a935a5d" exitCode=0 Oct 30 00:37:43.884688 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:43.884668 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a7ac6-75b958fbc4-dqgk7" event={"ID":"30bf357c-11ff-4fc6-bcfe-c776a3370b9f","Type":"ContainerDied","Data":"f2583b68cc5f20b95025592720f04fabb73f3a6d76b0377ba09a08df4a935a5d"} Oct 30 00:37:43.884796 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:43.884714 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a7ac6-75b958fbc4-dqgk7" event={"ID":"30bf357c-11ff-4fc6-bcfe-c776a3370b9f","Type":"ContainerDied","Data":"396f25f16fcb6997bb3cce4c8ff9716bf448ea71f6c4fdd763643096b75e928e"} Oct 30 00:37:43.884873 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:43.884797 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a7ac6-75b958fbc4-dqgk7" Oct 30 00:37:43.900921 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:43.900880 5001 scope.go:122] "RemoveContainer" containerID="14c8a467de7b1d26a1a5858f03e940d85a951858ef045595a7184ff2c77a57d8" Oct 30 00:37:43.901277 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:37:43.901254 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14c8a467de7b1d26a1a5858f03e940d85a951858ef045595a7184ff2c77a57d8\": container with ID starting with 14c8a467de7b1d26a1a5858f03e940d85a951858ef045595a7184ff2c77a57d8 not found: ID does not exist" containerID="14c8a467de7b1d26a1a5858f03e940d85a951858ef045595a7184ff2c77a57d8" Oct 30 00:37:43.901372 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:43.901289 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14c8a467de7b1d26a1a5858f03e940d85a951858ef045595a7184ff2c77a57d8"} err="failed to get container status \"14c8a467de7b1d26a1a5858f03e940d85a951858ef045595a7184ff2c77a57d8\": rpc error: code = NotFound desc = could not find container \"14c8a467de7b1d26a1a5858f03e940d85a951858ef045595a7184ff2c77a57d8\": container with ID starting with 14c8a467de7b1d26a1a5858f03e940d85a951858ef045595a7184ff2c77a57d8 not found: ID does not exist" Oct 30 00:37:43.901372 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:43.901314 5001 scope.go:122] "RemoveContainer" containerID="f2583b68cc5f20b95025592720f04fabb73f3a6d76b0377ba09a08df4a935a5d" Oct 30 00:37:43.914695 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:43.914676 5001 scope.go:122] "RemoveContainer" containerID="f2583b68cc5f20b95025592720f04fabb73f3a6d76b0377ba09a08df4a935a5d" Oct 30 00:37:43.915095 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:37:43.915071 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2583b68cc5f20b95025592720f04fabb73f3a6d76b0377ba09a08df4a935a5d\": container with ID starting with f2583b68cc5f20b95025592720f04fabb73f3a6d76b0377ba09a08df4a935a5d not found: ID does not exist" containerID="f2583b68cc5f20b95025592720f04fabb73f3a6d76b0377ba09a08df4a935a5d" Oct 30 00:37:43.915201 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:43.915101 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2583b68cc5f20b95025592720f04fabb73f3a6d76b0377ba09a08df4a935a5d"} err="failed to get container status \"f2583b68cc5f20b95025592720f04fabb73f3a6d76b0377ba09a08df4a935a5d\": rpc error: code = NotFound desc = could not find container \"f2583b68cc5f20b95025592720f04fabb73f3a6d76b0377ba09a08df4a935a5d\": container with ID starting with f2583b68cc5f20b95025592720f04fabb73f3a6d76b0377ba09a08df4a935a5d not found: ID does not exist" Oct 30 00:37:43.957942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:43.957690 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7a7ac6-75b958fbc4-dqgk7"] Oct 30 00:37:44.012999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:44.012963 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7a7ac6-75b958fbc4-dqgk7"] Oct 30 00:37:44.065934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:44.063400 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-7a7ac6-68f8c58595-vb58b"] Oct 30 00:37:44.121048 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:44.120958 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-7a7ac6-68f8c58595-vb58b"] Oct 30 00:37:44.167939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:44.164966 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e7fdaac-cp4qd"] Oct 30 00:37:44.193175 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:44.193137 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e7fdaac-cp4qd"] Oct 30 00:37:44.616947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:44.593573 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a7ac6-c9sl2"] Oct 30 00:37:44.660510 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:44.660465 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a7ac6-c9sl2"] Oct 30 00:37:44.921255 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:44.921182 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30bf357c-11ff-4fc6-bcfe-c776a3370b9f" path="/var/lib/kubelet/pods/30bf357c-11ff-4fc6-bcfe-c776a3370b9f/volumes" Oct 30 00:37:44.921689 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:44.921668 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="776a2d19-ec0d-4868-ac0c-e22a7c4b6de0" path="/var/lib/kubelet/pods/776a2d19-ec0d-4868-ac0c-e22a7c4b6de0/volumes" Oct 30 00:37:44.922249 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:44.922228 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="779d2d08-dbfe-4dd0-896f-25213ab7c12f" path="/var/lib/kubelet/pods/779d2d08-dbfe-4dd0-896f-25213ab7c12f/volumes" Oct 30 00:37:44.923041 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:44.923025 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3de6f63-a674-4c76-ace4-55b3c9f2cc12" path="/var/lib/kubelet/pods/b3de6f63-a674-4c76-ace4-55b3c9f2cc12/volumes" Oct 30 00:37:46.735154 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:46.732718 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn"] Oct 30 00:37:46.735154 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:46.733013 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" podUID="4deafe88-91e1-4878-9203-e28de81ce261" containerName="istio-proxy" containerID="cri-o://977bc8320a7d587e835c0ecebace3f3e8744acd49deb8a2609f106465549c600" gracePeriod=30 Oct 30 00:37:47.324416 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.324382 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ecb34a-d62vd"] Oct 30 00:37:47.325154 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.325133 5001 cpu_manager.go:414] "containerMap: removing container" podUID="776a2d19-ec0d-4868-ac0c-e22a7c4b6de0" containerName="payload-pre-processing" Oct 30 00:37:47.325154 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.325153 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="776a2d19-ec0d-4868-ac0c-e22a7c4b6de0" containerName="payload-pre-processing" Oct 30 00:37:47.325350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.325176 5001 cpu_manager.go:414] "containerMap: removing container" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" Oct 30 00:37:47.325350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.325184 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" Oct 30 00:37:47.325350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.325193 5001 cpu_manager.go:414] "containerMap: removing container" podUID="30bf357c-11ff-4fc6-bcfe-c776a3370b9f" containerName="payload-processing" Oct 30 00:37:47.325350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.325201 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30bf357c-11ff-4fc6-bcfe-c776a3370b9f" containerName="payload-processing" Oct 30 00:37:47.325350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.325241 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" Oct 30 00:37:47.325350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.325249 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" Oct 30 00:37:47.325350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.325262 5001 cpu_manager.go:414] "containerMap: removing container" podUID="67367a3a-0855-4843-b316-666d2654e6da" containerName="revoke-keys" Oct 30 00:37:47.325350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.325269 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67367a3a-0855-4843-b316-666d2654e6da" containerName="revoke-keys" Oct 30 00:37:47.325350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.325284 5001 cpu_manager.go:414] "containerMap: removing container" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" Oct 30 00:37:47.325350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.325291 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" Oct 30 00:37:47.325350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.325315 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d51bbbf9-d605-49bb-ae09-426c7a5622c0" containerName="revoke-keys" Oct 30 00:37:47.325350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.325323 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d51bbbf9-d605-49bb-ae09-426c7a5622c0" containerName="revoke-keys" Oct 30 00:37:47.325350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.325346 5001 cpu_manager.go:414] "containerMap: removing container" podUID="da718472-0862-4f25-8043-057c7fb59d17" containerName="revoke-keys" Oct 30 00:37:47.325350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.325355 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da718472-0862-4f25-8043-057c7fb59d17" containerName="revoke-keys" Oct 30 00:37:47.326137 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.325377 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f86ca908-fc4f-4c3a-a519-5d8f928576e3" containerName="revoke-keys" Oct 30 00:37:47.326137 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.325384 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f86ca908-fc4f-4c3a-a519-5d8f928576e3" containerName="revoke-keys" Oct 30 00:37:47.326137 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.325562 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="776a2d19-ec0d-4868-ac0c-e22a7c4b6de0" containerName="payload-pre-processing" Oct 30 00:37:47.326137 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.325580 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f0cfc05-3778-4a07-b822-cffc4d94ddb6" containerName="main" Oct 30 00:37:47.326137 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.325593 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="30bf357c-11ff-4fc6-bcfe-c776a3370b9f" containerName="payload-processing" Oct 30 00:37:47.326137 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.325609 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6cd17d2-ab47-4b1a-a605-f62ea23d4f5f" containerName="main" Oct 30 00:37:47.326137 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.325618 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="67367a3a-0855-4843-b316-666d2654e6da" containerName="revoke-keys" Oct 30 00:37:47.326137 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.325629 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="dde1d5db-cea4-4020-b843-2321a1e346c4" containerName="main" Oct 30 00:37:47.326137 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.325640 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d51bbbf9-d605-49bb-ae09-426c7a5622c0" containerName="revoke-keys" Oct 30 00:37:47.326137 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.325650 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="da718472-0862-4f25-8043-057c7fb59d17" containerName="revoke-keys" Oct 30 00:37:47.326137 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.325663 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f86ca908-fc4f-4c3a-a519-5d8f928576e3" containerName="revoke-keys" Oct 30 00:37:47.326669 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:37:47.326429 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:37:47.331134 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.331105 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ecb34a-d62vd" Oct 30 00:37:47.350729 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.350704 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ecb34a-d62vd"] Oct 30 00:37:47.353369 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.353344 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:37:47.353498 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.353365 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:37:47.353498 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.353368 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:37:47.353498 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.353468 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:37:47.458362 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.458332 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92fb1547-83d7-4b71-884d-b9fde3d388ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ecb34a-d62vd\" (UID: \"92fb1547-83d7-4b71-884d-b9fde3d388ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ecb34a-d62vd" Oct 30 00:37:47.458519 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.458401 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92fb1547-83d7-4b71-884d-b9fde3d388ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ecb34a-d62vd\" (UID: \"92fb1547-83d7-4b71-884d-b9fde3d388ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ecb34a-d62vd" Oct 30 00:37:47.458567 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.458534 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhjlh\" (UniqueName: \"kubernetes.io/projected/92fb1547-83d7-4b71-884d-b9fde3d388ec-kube-api-access-rhjlh\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ecb34a-d62vd\" (UID: \"92fb1547-83d7-4b71-884d-b9fde3d388ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ecb34a-d62vd" Oct 30 00:37:47.559454 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.559424 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92fb1547-83d7-4b71-884d-b9fde3d388ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ecb34a-d62vd\" (UID: \"92fb1547-83d7-4b71-884d-b9fde3d388ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ecb34a-d62vd" Oct 30 00:37:47.559619 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.559490 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92fb1547-83d7-4b71-884d-b9fde3d388ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ecb34a-d62vd\" (UID: \"92fb1547-83d7-4b71-884d-b9fde3d388ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ecb34a-d62vd" Oct 30 00:37:47.559619 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.559552 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhjlh\" (UniqueName: \"kubernetes.io/projected/92fb1547-83d7-4b71-884d-b9fde3d388ec-kube-api-access-rhjlh\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ecb34a-d62vd\" (UID: \"92fb1547-83d7-4b71-884d-b9fde3d388ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ecb34a-d62vd" Oct 30 00:37:47.560200 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.560172 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92fb1547-83d7-4b71-884d-b9fde3d388ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ecb34a-d62vd\" (UID: \"92fb1547-83d7-4b71-884d-b9fde3d388ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ecb34a-d62vd" Oct 30 00:37:47.561742 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.561714 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92fb1547-83d7-4b71-884d-b9fde3d388ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ecb34a-d62vd\" (UID: \"92fb1547-83d7-4b71-884d-b9fde3d388ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ecb34a-d62vd" Oct 30 00:37:47.583685 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.583628 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhjlh\" (UniqueName: \"kubernetes.io/projected/92fb1547-83d7-4b71-884d-b9fde3d388ec-kube-api-access-rhjlh\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ecb34a-d62vd\" (UID: \"92fb1547-83d7-4b71-884d-b9fde3d388ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ecb34a-d62vd" Oct 30 00:37:47.645947 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.645895 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ecb34a-d62vd" Oct 30 00:37:47.990141 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:47.990092 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ecb34a-d62vd"] Oct 30 00:37:48.089285 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:48.089251 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ecb34a-d62vd" event={"ID":"92fb1547-83d7-4b71-884d-b9fde3d388ec","Type":"ContainerStarted","Data":"34c921cd598287ec523880092975144395d137195d7e096d7f97a390c90a0028"} Oct 30 00:37:48.811028 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:48.809781 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91306600-8pxtv"] Oct 30 00:37:48.816155 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:48.814897 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91306600-8pxtv" Oct 30 00:37:48.847089 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:48.847056 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91306600-8pxtv"] Oct 30 00:37:48.853220 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:48.853189 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl"] Oct 30 00:37:48.901766 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:48.901727 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kmfb\" (UniqueName: \"kubernetes.io/projected/ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a-kube-api-access-4kmfb\") pod \"maas-api-revoke-sub-91306600-8pxtv\" (UID: \"ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91306600-8pxtv" Oct 30 00:37:48.901906 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:48.901796 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91306600-8pxtv\" (UID: \"ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91306600-8pxtv" Oct 30 00:37:48.901906 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:48.901819 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91306600-8pxtv\" (UID: \"ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91306600-8pxtv" Oct 30 00:37:49.003824 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:49.003789 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kmfb\" (UniqueName: \"kubernetes.io/projected/ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a-kube-api-access-4kmfb\") pod \"maas-api-revoke-sub-91306600-8pxtv\" (UID: \"ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91306600-8pxtv" Oct 30 00:37:49.004313 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:49.003840 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91306600-8pxtv\" (UID: \"ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91306600-8pxtv" Oct 30 00:37:49.004313 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:49.003868 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91306600-8pxtv\" (UID: \"ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91306600-8pxtv" Oct 30 00:37:49.004641 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:49.004613 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91306600-8pxtv\" (UID: \"ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91306600-8pxtv" Oct 30 00:37:49.006363 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:49.006337 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91306600-8pxtv\" (UID: \"ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91306600-8pxtv" Oct 30 00:37:49.046954 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:49.046768 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kmfb\" (UniqueName: \"kubernetes.io/projected/ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a-kube-api-access-4kmfb\") pod \"maas-api-revoke-sub-91306600-8pxtv\" (UID: \"ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91306600-8pxtv" Oct 30 00:37:49.140889 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:49.140434 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91306600-8pxtv" Oct 30 00:37:49.174819 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:49.173729 5001 generic.go:356] "Generic (PLEG): container finished" podID="92fb1547-83d7-4b71-884d-b9fde3d388ec" containerID="65d572919649102b99ec2f669343833b0098448e29924854c9ce5ec4b6ee8ddc" exitCode=0 Oct 30 00:37:49.174819 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:49.174280 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ecb34a-d62vd" event={"ID":"92fb1547-83d7-4b71-884d-b9fde3d388ec","Type":"ContainerDied","Data":"65d572919649102b99ec2f669343833b0098448e29924854c9ce5ec4b6ee8ddc"} Oct 30 00:37:49.293839 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:49.285585 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn"] Oct 30 00:37:49.547639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:49.547601 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-29f3ca-dvs48"] Oct 30 00:37:49.552307 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:49.552282 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-29f3ca-dvs48" Oct 30 00:37:49.647959 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:49.646564 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91306600-8pxtv"] Oct 30 00:37:49.647959 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:49.647441 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mzl2\" (UniqueName: \"kubernetes.io/projected/a76339ab-86f8-4e69-8844-730227c58fd7-kube-api-access-8mzl2\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-29f3ca-dvs48\" (UID: \"a76339ab-86f8-4e69-8844-730227c58fd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-29f3ca-dvs48" Oct 30 00:37:49.647959 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:49.647594 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a76339ab-86f8-4e69-8844-730227c58fd7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-29f3ca-dvs48\" (UID: \"a76339ab-86f8-4e69-8844-730227c58fd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-29f3ca-dvs48" Oct 30 00:37:49.647959 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:49.647719 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a76339ab-86f8-4e69-8844-730227c58fd7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-29f3ca-dvs48\" (UID: \"a76339ab-86f8-4e69-8844-730227c58fd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-29f3ca-dvs48" Oct 30 00:37:49.686642 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:49.686483 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-29f3ca-dvs48"] Oct 30 00:37:49.755095 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:49.753221 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a76339ab-86f8-4e69-8844-730227c58fd7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-29f3ca-dvs48\" (UID: \"a76339ab-86f8-4e69-8844-730227c58fd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-29f3ca-dvs48" Oct 30 00:37:49.755095 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:49.753330 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a76339ab-86f8-4e69-8844-730227c58fd7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-29f3ca-dvs48\" (UID: \"a76339ab-86f8-4e69-8844-730227c58fd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-29f3ca-dvs48" Oct 30 00:37:49.755095 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:49.753401 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mzl2\" (UniqueName: \"kubernetes.io/projected/a76339ab-86f8-4e69-8844-730227c58fd7-kube-api-access-8mzl2\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-29f3ca-dvs48\" (UID: \"a76339ab-86f8-4e69-8844-730227c58fd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-29f3ca-dvs48" Oct 30 00:37:49.755095 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:49.754259 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a76339ab-86f8-4e69-8844-730227c58fd7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-29f3ca-dvs48\" (UID: \"a76339ab-86f8-4e69-8844-730227c58fd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-29f3ca-dvs48" Oct 30 00:37:49.757659 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:49.757631 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a76339ab-86f8-4e69-8844-730227c58fd7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-29f3ca-dvs48\" (UID: \"a76339ab-86f8-4e69-8844-730227c58fd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-29f3ca-dvs48" Oct 30 00:37:49.784345 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:49.784310 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mzl2\" (UniqueName: \"kubernetes.io/projected/a76339ab-86f8-4e69-8844-730227c58fd7-kube-api-access-8mzl2\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-29f3ca-dvs48\" (UID: \"a76339ab-86f8-4e69-8844-730227c58fd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-29f3ca-dvs48" Oct 30 00:37:49.991708 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:49.990139 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-29f3ca-dvs48" Oct 30 00:37:50.189840 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:50.189522 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91306600-8pxtv" event={"ID":"ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a","Type":"ContainerStarted","Data":"c0a39699e2436c4542078ef46ff13f9a9e1963ed40dc554a05fa7b20c770dd65"} Oct 30 00:37:50.192935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:50.192067 5001 generic.go:356] "Generic (PLEG): container finished" podID="d50b1c24-43df-4ca1-b869-5e1d0d9d67ce" containerID="01a9dd7eecaab38e0fa624b7039a977439b2c4631fe839b9a14542329c833bc5" exitCode=2 Oct 30 00:37:50.192935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:50.192408 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" event={"ID":"d50b1c24-43df-4ca1-b869-5e1d0d9d67ce","Type":"ContainerDied","Data":"01a9dd7eecaab38e0fa624b7039a977439b2c4631fe839b9a14542329c833bc5"} Oct 30 00:37:50.192935 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:50.192931 5001 scope.go:122] "RemoveContainer" containerID="01a9dd7eecaab38e0fa624b7039a977439b2c4631fe839b9a14542329c833bc5" Oct 30 00:37:50.195097 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:50.195055 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:37:50.195202 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:50.195146 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:37:50.398008 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:37:50.397972 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76339ab_86f8_4e69_8844_730227c58fd7.slice/crio-3889bcbf8f0d0963003bc4611ec2c65d2e6987c4ab3eabc20061af5cd3bf8d2b WatchSource:0}: Error finding container 3889bcbf8f0d0963003bc4611ec2c65d2e6987c4ab3eabc20061af5cd3bf8d2b: Status 404 returned error can't find the container with id 3889bcbf8f0d0963003bc4611ec2c65d2e6987c4ab3eabc20061af5cd3bf8d2b Oct 30 00:37:50.399431 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:50.399406 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-29f3ca-dvs48"] Oct 30 00:37:50.424135 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:50.424111 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ecb34a-d62vd" Oct 30 00:37:50.477494 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:50.477460 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92fb1547-83d7-4b71-884d-b9fde3d388ec-kube-api-access-rhjlh\" (UniqueName: \"kubernetes.io/projected/92fb1547-83d7-4b71-884d-b9fde3d388ec-kube-api-access-rhjlh\") pod \"92fb1547-83d7-4b71-884d-b9fde3d388ec\" (UID: \"92fb1547-83d7-4b71-884d-b9fde3d388ec\") " Oct 30 00:37:50.477666 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:50.477644 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92fb1547-83d7-4b71-884d-b9fde3d388ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92fb1547-83d7-4b71-884d-b9fde3d388ec-maas-api-xks-ca\") pod \"92fb1547-83d7-4b71-884d-b9fde3d388ec\" (UID: \"92fb1547-83d7-4b71-884d-b9fde3d388ec\") " Oct 30 00:37:50.477762 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:50.477742 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92fb1547-83d7-4b71-884d-b9fde3d388ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92fb1547-83d7-4b71-884d-b9fde3d388ec-maas-api-openshift-ca\") pod \"92fb1547-83d7-4b71-884d-b9fde3d388ec\" (UID: \"92fb1547-83d7-4b71-884d-b9fde3d388ec\") " Oct 30 00:37:50.478182 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:50.478153 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92fb1547-83d7-4b71-884d-b9fde3d388ec-maas-api-openshift-ca" pod "92fb1547-83d7-4b71-884d-b9fde3d388ec" (UID: "92fb1547-83d7-4b71-884d-b9fde3d388ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:37:50.478361 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:50.478339 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92fb1547-83d7-4b71-884d-b9fde3d388ec-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:50.479869 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:50.479841 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92fb1547-83d7-4b71-884d-b9fde3d388ec-kube-api-access-rhjlh" pod "92fb1547-83d7-4b71-884d-b9fde3d388ec" (UID: "92fb1547-83d7-4b71-884d-b9fde3d388ec"). InnerVolumeSpecName "kube-api-access-rhjlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:50.480016 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:50.479885 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92fb1547-83d7-4b71-884d-b9fde3d388ec-maas-api-xks-ca" pod "92fb1547-83d7-4b71-884d-b9fde3d388ec" (UID: "92fb1547-83d7-4b71-884d-b9fde3d388ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:37:50.579695 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:50.579603 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92fb1547-83d7-4b71-884d-b9fde3d388ec-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:50.579695 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:50.579638 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhjlh\" (UniqueName: \"kubernetes.io/projected/92fb1547-83d7-4b71-884d-b9fde3d388ec-kube-api-access-rhjlh\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:51.208270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:51.208161 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-29f3ca-dvs48" event={"ID":"a76339ab-86f8-4e69-8844-730227c58fd7","Type":"ContainerStarted","Data":"22ef92cd095331935e272ea01f9eac7ef7e1aecee8d8cbd0f5556096708c4d90"} Oct 30 00:37:51.208846 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:51.208803 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-29f3ca-dvs48" event={"ID":"a76339ab-86f8-4e69-8844-730227c58fd7","Type":"ContainerStarted","Data":"3889bcbf8f0d0963003bc4611ec2c65d2e6987c4ab3eabc20061af5cd3bf8d2b"} Oct 30 00:37:51.210814 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:51.210783 5001 generic.go:356] "Generic (PLEG): container finished" podID="ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a" containerID="5678865fdbd79c463f07e2c709853326872b5968afd921a126277a8ab5f587fd" exitCode=0 Oct 30 00:37:51.210961 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:51.210880 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91306600-8pxtv" event={"ID":"ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a","Type":"ContainerDied","Data":"5678865fdbd79c463f07e2c709853326872b5968afd921a126277a8ab5f587fd"} Oct 30 00:37:51.213740 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:51.213524 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" event={"ID":"d50b1c24-43df-4ca1-b869-5e1d0d9d67ce","Type":"ContainerStarted","Data":"c296870528374b9f6c1db5c37f0dbd8739d92b13759872338eb4fa1c320bb235"} Oct 30 00:37:51.213838 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:51.213804 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" Oct 30 00:37:51.215391 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:51.215367 5001 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-2z4vm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:37:51.215501 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:51.215429 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" podUID="d50b1c24-43df-4ca1-b869-5e1d0d9d67ce" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:37:51.216453 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:51.216430 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ecb34a-d62vd" event={"ID":"92fb1547-83d7-4b71-884d-b9fde3d388ec","Type":"ContainerDied","Data":"34c921cd598287ec523880092975144395d137195d7e096d7f97a390c90a0028"} Oct 30 00:37:51.216453 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:51.216448 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ecb34a-d62vd" Oct 30 00:37:51.216598 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:51.216474 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34c921cd598287ec523880092975144395d137195d7e096d7f97a390c90a0028" Oct 30 00:37:51.293850 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:51.293788 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-29f3ca-dvs48" podStartSLOduration=2.293770663 podStartE2EDuration="2.293770663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:37:51.292383281 +0000 UTC m=+1986.497884464" watchObservedRunningTime="2026-10-30 00:37:51.293770663 +0000 UTC m=+1986.499271845" Oct 30 00:37:51.851629 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:51.851590 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c1eeb6-7d97d86c79-zmp4b" podUID="2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:37:52.227960 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.227863 5001 generic.go:356] "Generic (PLEG): container finished" podID="a76339ab-86f8-4e69-8844-730227c58fd7" containerID="22ef92cd095331935e272ea01f9eac7ef7e1aecee8d8cbd0f5556096708c4d90" exitCode=0 Oct 30 00:37:52.228418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.227950 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-29f3ca-dvs48" event={"ID":"a76339ab-86f8-4e69-8844-730227c58fd7","Type":"ContainerDied","Data":"22ef92cd095331935e272ea01f9eac7ef7e1aecee8d8cbd0f5556096708c4d90"} Oct 30 00:37:52.230131 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.230106 5001 generic.go:356] "Generic (PLEG): container finished" podID="4deafe88-91e1-4878-9203-e28de81ce261" containerID="977bc8320a7d587e835c0ecebace3f3e8744acd49deb8a2609f106465549c600" exitCode=0 Oct 30 00:37:52.230261 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.230169 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" event={"ID":"4deafe88-91e1-4878-9203-e28de81ce261","Type":"ContainerDied","Data":"977bc8320a7d587e835c0ecebace3f3e8744acd49deb8a2609f106465549c600"} Oct 30 00:37:52.230261 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.230224 5001 scope.go:122] "RemoveContainer" containerID="933325317e94c64b0c6c337585297e652db8cab8364924dd9ede8a26aa107a3d" Oct 30 00:37:52.231110 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.231092 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2z4vm" Oct 30 00:37:52.511616 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.511583 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn"] Oct 30 00:37:52.511803 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.511760 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" podUID="74a414d8-654e-4a80-bfaf-1ea96f68ea8e" containerName="istio-proxy" containerID="cri-o://fbe80941790a78584998bb0942f567da5bb31636353fd30b0cdcf7367a5595a0" gracePeriod=30 Oct 30 00:37:52.564139 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.552450 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx"] Oct 30 00:37:52.564139 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.556107 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" podUID="ad5882d9-90e8-400f-90d5-d4ba5d8da5fc" containerName="istio-proxy" containerID="cri-o://cf90f41004529a7c7f998958db88becac65c20dfc62fc7a12df95ca23f4d05fa" gracePeriod=30 Oct 30 00:37:52.578593 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.578552 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf"] Oct 30 00:37:52.578895 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.578867 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" podUID="d34b94b7-c280-4d34-a4b3-5ae295c595cb" containerName="istio-proxy" containerID="cri-o://6b697ac96104253924b98717b1255c234b39e296acada0ae409b63ea60d3159c" gracePeriod=30 Oct 30 00:37:52.596943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.595252 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91306600-8pxtv" Oct 30 00:37:52.598329 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.597710 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687"] Oct 30 00:37:52.598329 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.597941 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" podUID="ea34fb77-6b08-4939-b1c0-9ba33e8afd0f" containerName="istio-proxy" containerID="cri-o://2424084982f40ddcac1405610a72c7eca9b7f6985742ea3921953c90f8649a98" gracePeriod=30 Oct 30 00:37:52.618518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.618497 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:37:52.722988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.722382 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4deafe88-91e1-4878-9203-e28de81ce261-kube-api-access-j645z\" (UniqueName: \"kubernetes.io/projected/4deafe88-91e1-4878-9203-e28de81ce261-kube-api-access-j645z\") pod \"4deafe88-91e1-4878-9203-e28de81ce261\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " Oct 30 00:37:52.722988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.722440 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4deafe88-91e1-4878-9203-e28de81ce261-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4deafe88-91e1-4878-9203-e28de81ce261-istiod-ca-cert\") pod \"4deafe88-91e1-4878-9203-e28de81ce261\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " Oct 30 00:37:52.722988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.722478 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4deafe88-91e1-4878-9203-e28de81ce261-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4deafe88-91e1-4878-9203-e28de81ce261-istio-podinfo\") pod \"4deafe88-91e1-4878-9203-e28de81ce261\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " Oct 30 00:37:52.722988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.722511 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a-kube-api-access-4kmfb\" (UniqueName: \"kubernetes.io/projected/ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a-kube-api-access-4kmfb\") pod \"ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a\" (UID: \"ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a\") " Oct 30 00:37:52.722988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.722573 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-workload-socket\") pod \"4deafe88-91e1-4878-9203-e28de81ce261\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " Oct 30 00:37:52.722988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.722632 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a-maas-api-openshift-ca\") pod \"ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a\" (UID: \"ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a\") " Oct 30 00:37:52.722988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.722679 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-credential-socket\") pod \"4deafe88-91e1-4878-9203-e28de81ce261\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " Oct 30 00:37:52.722988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.722721 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4deafe88-91e1-4878-9203-e28de81ce261-istio-token\" (UniqueName: \"kubernetes.io/projected/4deafe88-91e1-4878-9203-e28de81ce261-istio-token\") pod \"4deafe88-91e1-4878-9203-e28de81ce261\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " Oct 30 00:37:52.722988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.722759 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-workload-certs\") pod \"4deafe88-91e1-4878-9203-e28de81ce261\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " Oct 30 00:37:52.722988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.722849 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-istio-data\") pod \"4deafe88-91e1-4878-9203-e28de81ce261\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " Oct 30 00:37:52.722988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.722981 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a-maas-api-xks-ca\") pod \"ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a\" (UID: \"ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a\") " Oct 30 00:37:52.723836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.723209 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-istio-envoy\") pod \"4deafe88-91e1-4878-9203-e28de81ce261\" (UID: \"4deafe88-91e1-4878-9203-e28de81ce261\") " Oct 30 00:37:52.736978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.727372 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a-maas-api-openshift-ca" pod "ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a" (UID: "ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:37:52.736978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.727492 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-credential-socket" pod "4deafe88-91e1-4878-9203-e28de81ce261" (UID: "4deafe88-91e1-4878-9203-e28de81ce261"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:52.736978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.729252 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-istio-data" pod "4deafe88-91e1-4878-9203-e28de81ce261" (UID: "4deafe88-91e1-4878-9203-e28de81ce261"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:52.736978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.729662 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-workload-certs" pod "4deafe88-91e1-4878-9203-e28de81ce261" (UID: "4deafe88-91e1-4878-9203-e28de81ce261"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:52.736978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.731236 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-workload-socket" pod "4deafe88-91e1-4878-9203-e28de81ce261" (UID: "4deafe88-91e1-4878-9203-e28de81ce261"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:52.736978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.731423 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4deafe88-91e1-4878-9203-e28de81ce261-istiod-ca-cert" pod "4deafe88-91e1-4878-9203-e28de81ce261" (UID: "4deafe88-91e1-4878-9203-e28de81ce261"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:37:52.736978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.731611 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4deafe88-91e1-4878-9203-e28de81ce261-istio-token" pod "4deafe88-91e1-4878-9203-e28de81ce261" (UID: "4deafe88-91e1-4878-9203-e28de81ce261"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:52.736978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.734467 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4deafe88-91e1-4878-9203-e28de81ce261-kube-api-access-j645z" pod "4deafe88-91e1-4878-9203-e28de81ce261" (UID: "4deafe88-91e1-4878-9203-e28de81ce261"). InnerVolumeSpecName "kube-api-access-j645z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:52.736978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.734616 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a-maas-api-xks-ca" pod "ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a" (UID: "ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:37:52.736978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.734640 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-istio-envoy" pod "4deafe88-91e1-4878-9203-e28de81ce261" (UID: "4deafe88-91e1-4878-9203-e28de81ce261"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:52.736978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.735077 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4deafe88-91e1-4878-9203-e28de81ce261-istio-podinfo" pod "4deafe88-91e1-4878-9203-e28de81ce261" (UID: "4deafe88-91e1-4878-9203-e28de81ce261"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:37:52.736978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.736533 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a-kube-api-access-4kmfb" pod "ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a" (UID: "ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a"). InnerVolumeSpecName "kube-api-access-4kmfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:52.826709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.825356 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j645z\" (UniqueName: \"kubernetes.io/projected/4deafe88-91e1-4878-9203-e28de81ce261-kube-api-access-j645z\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:52.826709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.825403 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4deafe88-91e1-4878-9203-e28de81ce261-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:52.826709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.825414 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4deafe88-91e1-4878-9203-e28de81ce261-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:52.826709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.825425 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kmfb\" (UniqueName: \"kubernetes.io/projected/ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a-kube-api-access-4kmfb\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:52.826709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.825435 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:52.826709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.825444 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:52.826709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.825453 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:52.826709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.825461 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4deafe88-91e1-4878-9203-e28de81ce261-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:52.826709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.825470 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:52.826709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.825478 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:52.826709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.825486 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:52.826709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:52.825494 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4deafe88-91e1-4878-9203-e28de81ce261-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:53.267942 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:53.267896 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91306600-8pxtv" Oct 30 00:37:53.268389 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:53.267933 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91306600-8pxtv" event={"ID":"ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a","Type":"ContainerDied","Data":"c0a39699e2436c4542078ef46ff13f9a9e1963ed40dc554a05fa7b20c770dd65"} Oct 30 00:37:53.268389 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:53.267990 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0a39699e2436c4542078ef46ff13f9a9e1963ed40dc554a05fa7b20c770dd65" Oct 30 00:37:53.269886 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:53.269859 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" Oct 30 00:37:53.270050 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:53.269924 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn" event={"ID":"4deafe88-91e1-4878-9203-e28de81ce261","Type":"ContainerDied","Data":"78828a98e504181c16613517d0590c654945b2f8b087c17c25ab476fc8c46f89"} Oct 30 00:37:53.270050 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:53.269978 5001 scope.go:122] "RemoveContainer" containerID="977bc8320a7d587e835c0ecebace3f3e8744acd49deb8a2609f106465549c600" Oct 30 00:37:53.331132 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:53.331099 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn"] Oct 30 00:37:53.342194 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:53.342165 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c1eeb6-openshift-default-84cf98656c-8zxfn"] Oct 30 00:37:53.451195 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:53.451172 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-29f3ca-dvs48" Oct 30 00:37:53.538493 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:53.538409 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a76339ab-86f8-4e69-8844-730227c58fd7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a76339ab-86f8-4e69-8844-730227c58fd7-maas-api-openshift-ca\") pod \"a76339ab-86f8-4e69-8844-730227c58fd7\" (UID: \"a76339ab-86f8-4e69-8844-730227c58fd7\") " Oct 30 00:37:53.538652 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:53.538527 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a76339ab-86f8-4e69-8844-730227c58fd7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a76339ab-86f8-4e69-8844-730227c58fd7-maas-api-xks-ca\") pod \"a76339ab-86f8-4e69-8844-730227c58fd7\" (UID: \"a76339ab-86f8-4e69-8844-730227c58fd7\") " Oct 30 00:37:53.538652 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:53.538639 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a76339ab-86f8-4e69-8844-730227c58fd7-kube-api-access-8mzl2\" (UniqueName: \"kubernetes.io/projected/a76339ab-86f8-4e69-8844-730227c58fd7-kube-api-access-8mzl2\") pod \"a76339ab-86f8-4e69-8844-730227c58fd7\" (UID: \"a76339ab-86f8-4e69-8844-730227c58fd7\") " Oct 30 00:37:53.538832 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:53.538805 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a76339ab-86f8-4e69-8844-730227c58fd7-maas-api-openshift-ca" pod "a76339ab-86f8-4e69-8844-730227c58fd7" (UID: "a76339ab-86f8-4e69-8844-730227c58fd7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:37:53.539442 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:53.539411 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a76339ab-86f8-4e69-8844-730227c58fd7-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:53.540843 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:53.540812 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a76339ab-86f8-4e69-8844-730227c58fd7-maas-api-xks-ca" pod "a76339ab-86f8-4e69-8844-730227c58fd7" (UID: "a76339ab-86f8-4e69-8844-730227c58fd7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:37:53.540970 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:53.540839 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a76339ab-86f8-4e69-8844-730227c58fd7-kube-api-access-8mzl2" pod "a76339ab-86f8-4e69-8844-730227c58fd7" (UID: "a76339ab-86f8-4e69-8844-730227c58fd7"). InnerVolumeSpecName "kube-api-access-8mzl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:53.640883 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:53.640842 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mzl2\" (UniqueName: \"kubernetes.io/projected/a76339ab-86f8-4e69-8844-730227c58fd7-kube-api-access-8mzl2\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:53.640883 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:53.640877 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a76339ab-86f8-4e69-8844-730227c58fd7-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:54.281107 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:54.281075 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-29f3ca-dvs48" Oct 30 00:37:54.281552 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:54.281130 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-29f3ca-dvs48" event={"ID":"a76339ab-86f8-4e69-8844-730227c58fd7","Type":"ContainerDied","Data":"3889bcbf8f0d0963003bc4611ec2c65d2e6987c4ab3eabc20061af5cd3bf8d2b"} Oct 30 00:37:54.281552 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:54.281178 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3889bcbf8f0d0963003bc4611ec2c65d2e6987c4ab3eabc20061af5cd3bf8d2b" Oct 30 00:37:54.925387 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:54.925345 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4deafe88-91e1-4878-9203-e28de81ce261" path="/var/lib/kubelet/pods/4deafe88-91e1-4878-9203-e28de81ce261/volumes" Oct 30 00:37:55.476179 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:55.476134 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:37:55.476561 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:55.476204 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" podUID="d34b94b7-c280-4d34-a4b3-5ae295c595cb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:37:57.028120 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:57.028086 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e2bb288-7nz5q"] Oct 30 00:37:57.037891 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:57.037851 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e2bb288-7nz5q"] Oct 30 00:37:57.785573 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:57.784633 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-ecb34a-7c4f79cf86-6l8nz"] Oct 30 00:37:57.804869 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:57.804835 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ecb34a-7ffdbf96c-8lgp2"] Oct 30 00:37:57.962493 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:57.962465 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:37:58.022821 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.021316 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istiod-ca-cert\") pod \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " Oct 30 00:37:58.022821 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.021402 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-credential-socket\") pod \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " Oct 30 00:37:58.022821 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.021446 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istio-envoy\") pod \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " Oct 30 00:37:58.022821 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.021508 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-kube-api-access-r2qb2\" (UniqueName: \"kubernetes.io/projected/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-kube-api-access-r2qb2\") pod \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " Oct 30 00:37:58.022821 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.021556 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istio-token\" (UniqueName: \"kubernetes.io/projected/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istio-token\") pod \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " Oct 30 00:37:58.022821 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.021786 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istio-podinfo\") pod \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " Oct 30 00:37:58.022821 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.021857 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istio-data\") pod \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " Oct 30 00:37:58.022821 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.021890 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-workload-socket\") pod \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " Oct 30 00:37:58.022821 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.021968 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-workload-certs\") pod \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\" (UID: \"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc\") " Oct 30 00:37:58.022821 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.022060 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-credential-socket" pod "ad5882d9-90e8-400f-90d5-d4ba5d8da5fc" (UID: "ad5882d9-90e8-400f-90d5-d4ba5d8da5fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.022821 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.022475 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istiod-ca-cert" pod "ad5882d9-90e8-400f-90d5-d4ba5d8da5fc" (UID: "ad5882d9-90e8-400f-90d5-d4ba5d8da5fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:37:58.023647 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.022952 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-workload-certs" pod "ad5882d9-90e8-400f-90d5-d4ba5d8da5fc" (UID: "ad5882d9-90e8-400f-90d5-d4ba5d8da5fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.023647 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.023581 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istio-data" pod "ad5882d9-90e8-400f-90d5-d4ba5d8da5fc" (UID: "ad5882d9-90e8-400f-90d5-d4ba5d8da5fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.025399 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.024192 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.025399 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.024475 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.025399 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.024492 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.028970 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.028793 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-workload-socket" pod "ad5882d9-90e8-400f-90d5-d4ba5d8da5fc" (UID: "ad5882d9-90e8-400f-90d5-d4ba5d8da5fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.083059 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.082192 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-kube-api-access-r2qb2" pod "ad5882d9-90e8-400f-90d5-d4ba5d8da5fc" (UID: "ad5882d9-90e8-400f-90d5-d4ba5d8da5fc"). InnerVolumeSpecName "kube-api-access-r2qb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:58.083984 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.083938 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istio-token" pod "ad5882d9-90e8-400f-90d5-d4ba5d8da5fc" (UID: "ad5882d9-90e8-400f-90d5-d4ba5d8da5fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:58.084274 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.084238 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istio-podinfo" pod "ad5882d9-90e8-400f-90d5-d4ba5d8da5fc" (UID: "ad5882d9-90e8-400f-90d5-d4ba5d8da5fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:37:58.089439 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.089399 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istio-envoy" pod "ad5882d9-90e8-400f-90d5-d4ba5d8da5fc" (UID: "ad5882d9-90e8-400f-90d5-d4ba5d8da5fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.091643 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.091616 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:37:58.127176 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.126897 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.127176 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.126940 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2qb2\" (UniqueName: \"kubernetes.io/projected/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-kube-api-access-r2qb2\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.127176 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.126952 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.127176 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.126960 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.127176 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.126968 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.127176 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.126978 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.148190 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.148158 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:37:58.229500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.228141 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istio-podinfo\") pod \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " Oct 30 00:37:58.229500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.228176 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d34b94b7-c280-4d34-a4b3-5ae295c595cb-kube-api-access-7ccqw\" (UniqueName: \"kubernetes.io/projected/d34b94b7-c280-4d34-a4b3-5ae295c595cb-kube-api-access-7ccqw\") pod \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " Oct 30 00:37:58.229500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.228203 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istiod-ca-cert\") pod \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " Oct 30 00:37:58.229500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.228232 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istio-envoy\") pod \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " Oct 30 00:37:58.229500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.228269 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istio-data\") pod \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " Oct 30 00:37:58.229500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.228293 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-credential-socket\") pod \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " Oct 30 00:37:58.229500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.228339 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-workload-socket\") pod \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " Oct 30 00:37:58.229500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.228367 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-kube-api-access-llrzt\" (UniqueName: \"kubernetes.io/projected/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-kube-api-access-llrzt\") pod \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " Oct 30 00:37:58.229500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.228407 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istio-envoy\") pod \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " Oct 30 00:37:58.229500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.228467 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istio-token\" (UniqueName: \"kubernetes.io/projected/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istio-token\") pod \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " Oct 30 00:37:58.229500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.228519 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istio-data\") pod \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " Oct 30 00:37:58.229500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.228555 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-credential-socket\") pod \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " Oct 30 00:37:58.229500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.228593 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istio-podinfo\") pod \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " Oct 30 00:37:58.229500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.228647 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istiod-ca-cert\") pod \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " Oct 30 00:37:58.229500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.228743 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-workload-certs\") pod \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " Oct 30 00:37:58.229500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.228828 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istio-token\" (UniqueName: \"kubernetes.io/projected/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istio-token\") pod \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " Oct 30 00:37:58.229500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.228896 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-workload-certs\") pod \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\" (UID: \"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f\") " Oct 30 00:37:58.229500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.229419 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-workload-socket" pod "ea34fb77-6b08-4939-b1c0-9ba33e8afd0f" (UID: "ea34fb77-6b08-4939-b1c0-9ba33e8afd0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.229500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.229459 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istio-data" pod "ea34fb77-6b08-4939-b1c0-9ba33e8afd0f" (UID: "ea34fb77-6b08-4939-b1c0-9ba33e8afd0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.230358 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.229358 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istiod-ca-cert" pod "ea34fb77-6b08-4939-b1c0-9ba33e8afd0f" (UID: "ea34fb77-6b08-4939-b1c0-9ba33e8afd0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:37:58.230358 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.229652 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-credential-socket" pod "ea34fb77-6b08-4939-b1c0-9ba33e8afd0f" (UID: "ea34fb77-6b08-4939-b1c0-9ba33e8afd0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.230358 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.229835 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istio-data" pod "d34b94b7-c280-4d34-a4b3-5ae295c595cb" (UID: "d34b94b7-c280-4d34-a4b3-5ae295c595cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.230531 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.230506 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-workload-socket\") pod \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\" (UID: \"d34b94b7-c280-4d34-a4b3-5ae295c595cb\") " Oct 30 00:37:58.231199 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.231181 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.231322 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.231204 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.231322 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.231219 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.231322 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.231234 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.231322 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.231248 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.231560 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.231498 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-workload-socket" pod "d34b94b7-c280-4d34-a4b3-5ae295c595cb" (UID: "d34b94b7-c280-4d34-a4b3-5ae295c595cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.231842 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.231766 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istiod-ca-cert" pod "d34b94b7-c280-4d34-a4b3-5ae295c595cb" (UID: "d34b94b7-c280-4d34-a4b3-5ae295c595cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:37:58.232111 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.232091 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-workload-certs" pod "d34b94b7-c280-4d34-a4b3-5ae295c595cb" (UID: "d34b94b7-c280-4d34-a4b3-5ae295c595cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.235533 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.235504 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-credential-socket" pod "d34b94b7-c280-4d34-a4b3-5ae295c595cb" (UID: "d34b94b7-c280-4d34-a4b3-5ae295c595cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.235728 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.235707 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istio-token" pod "ea34fb77-6b08-4939-b1c0-9ba33e8afd0f" (UID: "ea34fb77-6b08-4939-b1c0-9ba33e8afd0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:58.235903 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.235882 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-workload-certs" pod "ea34fb77-6b08-4939-b1c0-9ba33e8afd0f" (UID: "ea34fb77-6b08-4939-b1c0-9ba33e8afd0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.237375 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.237311 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istio-token" pod "d34b94b7-c280-4d34-a4b3-5ae295c595cb" (UID: "d34b94b7-c280-4d34-a4b3-5ae295c595cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:58.237375 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.237346 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-kube-api-access-llrzt" pod "ea34fb77-6b08-4939-b1c0-9ba33e8afd0f" (UID: "ea34fb77-6b08-4939-b1c0-9ba33e8afd0f"). InnerVolumeSpecName "kube-api-access-llrzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:58.237583 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.237389 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istio-podinfo" pod "d34b94b7-c280-4d34-a4b3-5ae295c595cb" (UID: "d34b94b7-c280-4d34-a4b3-5ae295c595cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:37:58.237745 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.237724 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istio-podinfo" pod "ea34fb77-6b08-4939-b1c0-9ba33e8afd0f" (UID: "ea34fb77-6b08-4939-b1c0-9ba33e8afd0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:37:58.237826 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.237741 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istio-envoy" pod "d34b94b7-c280-4d34-a4b3-5ae295c595cb" (UID: "d34b94b7-c280-4d34-a4b3-5ae295c595cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.237826 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.237811 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d34b94b7-c280-4d34-a4b3-5ae295c595cb-kube-api-access-7ccqw" pod "d34b94b7-c280-4d34-a4b3-5ae295c595cb" (UID: "d34b94b7-c280-4d34-a4b3-5ae295c595cb"). InnerVolumeSpecName "kube-api-access-7ccqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:58.238943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.238904 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istio-envoy" pod "ea34fb77-6b08-4939-b1c0-9ba33e8afd0f" (UID: "ea34fb77-6b08-4939-b1c0-9ba33e8afd0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.280734 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.280704 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:37:58.330282 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.329949 5001 generic.go:356] "Generic (PLEG): container finished" podID="ad5882d9-90e8-400f-90d5-d4ba5d8da5fc" containerID="cf90f41004529a7c7f998958db88becac65c20dfc62fc7a12df95ca23f4d05fa" exitCode=0 Oct 30 00:37:58.330282 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.330077 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" event={"ID":"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc","Type":"ContainerDied","Data":"cf90f41004529a7c7f998958db88becac65c20dfc62fc7a12df95ca23f4d05fa"} Oct 30 00:37:58.330282 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.330131 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" Oct 30 00:37:58.330282 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.330141 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx" event={"ID":"ad5882d9-90e8-400f-90d5-d4ba5d8da5fc","Type":"ContainerDied","Data":"a9c41186ebf661d7a0e35ca6699bcb3657725614e1b5d2e6202dcfab6f1b733f"} Oct 30 00:37:58.330282 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.330173 5001 scope.go:122] "RemoveContainer" containerID="cf90f41004529a7c7f998958db88becac65c20dfc62fc7a12df95ca23f4d05fa" Oct 30 00:37:58.331893 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.331864 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-credential-socket\") pod \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " Oct 30 00:37:58.332103 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.331962 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-workload-socket\") pod \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " Oct 30 00:37:58.332103 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.332020 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istio-data\") pod \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " Oct 30 00:37:58.332103 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.332090 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istio-token\" (UniqueName: \"kubernetes.io/projected/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istio-token\") pod \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " Oct 30 00:37:58.332403 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.332136 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-workload-certs\") pod \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " Oct 30 00:37:58.332403 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.332242 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-credential-socket" pod "74a414d8-654e-4a80-bfaf-1ea96f68ea8e" (UID: "74a414d8-654e-4a80-bfaf-1ea96f68ea8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.332403 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.332322 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istiod-ca-cert\") pod \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " Oct 30 00:37:58.332403 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.332370 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-kube-api-access-54658\" (UniqueName: \"kubernetes.io/projected/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-kube-api-access-54658\") pod \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " Oct 30 00:37:58.332621 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.332431 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istio-podinfo\") pod \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " Oct 30 00:37:58.332621 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.332450 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istio-data" pod "74a414d8-654e-4a80-bfaf-1ea96f68ea8e" (UID: "74a414d8-654e-4a80-bfaf-1ea96f68ea8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.332621 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.332590 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istio-envoy\") pod \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\" (UID: \"74a414d8-654e-4a80-bfaf-1ea96f68ea8e\") " Oct 30 00:37:58.332775 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.332662 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-workload-socket" pod "74a414d8-654e-4a80-bfaf-1ea96f68ea8e" (UID: "74a414d8-654e-4a80-bfaf-1ea96f68ea8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.333746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.332970 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istiod-ca-cert" pod "74a414d8-654e-4a80-bfaf-1ea96f68ea8e" (UID: "74a414d8-654e-4a80-bfaf-1ea96f68ea8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:37:58.333746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.333227 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.333746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.333249 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ccqw\" (UniqueName: \"kubernetes.io/projected/d34b94b7-c280-4d34-a4b3-5ae295c595cb-kube-api-access-7ccqw\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.333746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.333265 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.333746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.333279 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.333746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.333293 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llrzt\" (UniqueName: \"kubernetes.io/projected/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-kube-api-access-llrzt\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.333746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.333306 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.333746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.333319 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.333746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.333331 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.333746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.333345 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.333746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.333358 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d34b94b7-c280-4d34-a4b3-5ae295c595cb-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.333746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.333370 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.333746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.333383 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.333746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.333397 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.333746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.333410 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.333746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.333423 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.333746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.333435 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.333746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.333446 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34b94b7-c280-4d34-a4b3-5ae295c595cb-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.336976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.334969 5001 generic.go:356] "Generic (PLEG): container finished" podID="74a414d8-654e-4a80-bfaf-1ea96f68ea8e" containerID="fbe80941790a78584998bb0942f567da5bb31636353fd30b0cdcf7367a5595a0" exitCode=0 Oct 30 00:37:58.336976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.335170 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" event={"ID":"74a414d8-654e-4a80-bfaf-1ea96f68ea8e","Type":"ContainerDied","Data":"fbe80941790a78584998bb0942f567da5bb31636353fd30b0cdcf7367a5595a0"} Oct 30 00:37:58.336976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.335215 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" event={"ID":"74a414d8-654e-4a80-bfaf-1ea96f68ea8e","Type":"ContainerDied","Data":"e06d92a194de75b6a754c863aecdf0f098f7cc9a89eb03958462e66dac6b73f6"} Oct 30 00:37:58.336976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.335294 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn" Oct 30 00:37:58.336976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.336025 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istio-envoy" pod "74a414d8-654e-4a80-bfaf-1ea96f68ea8e" (UID: "74a414d8-654e-4a80-bfaf-1ea96f68ea8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.337718 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.337054 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istio-token" pod "74a414d8-654e-4a80-bfaf-1ea96f68ea8e" (UID: "74a414d8-654e-4a80-bfaf-1ea96f68ea8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:58.337718 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.337110 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-kube-api-access-54658" pod "74a414d8-654e-4a80-bfaf-1ea96f68ea8e" (UID: "74a414d8-654e-4a80-bfaf-1ea96f68ea8e"). InnerVolumeSpecName "kube-api-access-54658". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:58.337718 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.337428 5001 generic.go:356] "Generic (PLEG): container finished" podID="d34b94b7-c280-4d34-a4b3-5ae295c595cb" containerID="6b697ac96104253924b98717b1255c234b39e296acada0ae409b63ea60d3159c" exitCode=0 Oct 30 00:37:58.337718 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.337523 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" event={"ID":"d34b94b7-c280-4d34-a4b3-5ae295c595cb","Type":"ContainerDied","Data":"6b697ac96104253924b98717b1255c234b39e296acada0ae409b63ea60d3159c"} Oct 30 00:37:58.337718 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.337562 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" event={"ID":"d34b94b7-c280-4d34-a4b3-5ae295c595cb","Type":"ContainerDied","Data":"a55a8a92ce082cea8f9b043faafda9ced69565c3b10cb1c7a11c3ba674c389aa"} Oct 30 00:37:58.337718 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.337628 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf" Oct 30 00:37:58.338088 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.337865 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istio-podinfo" pod "74a414d8-654e-4a80-bfaf-1ea96f68ea8e" (UID: "74a414d8-654e-4a80-bfaf-1ea96f68ea8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:37:58.338293 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.338247 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-workload-certs" pod "74a414d8-654e-4a80-bfaf-1ea96f68ea8e" (UID: "74a414d8-654e-4a80-bfaf-1ea96f68ea8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:58.339881 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.339855 5001 generic.go:356] "Generic (PLEG): container finished" podID="ea34fb77-6b08-4939-b1c0-9ba33e8afd0f" containerID="2424084982f40ddcac1405610a72c7eca9b7f6985742ea3921953c90f8649a98" exitCode=0 Oct 30 00:37:58.340095 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.340072 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" event={"ID":"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f","Type":"ContainerDied","Data":"2424084982f40ddcac1405610a72c7eca9b7f6985742ea3921953c90f8649a98"} Oct 30 00:37:58.340205 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.340121 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" event={"ID":"ea34fb77-6b08-4939-b1c0-9ba33e8afd0f","Type":"ContainerDied","Data":"167b1993be7149a3ce8fb54ddeb3b980edb4ce7230d83c07711d8d3a765c43f0"} Oct 30 00:37:58.340205 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.340193 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687" Oct 30 00:37:58.360736 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.359958 5001 scope.go:122] "RemoveContainer" containerID="cf90f41004529a7c7f998958db88becac65c20dfc62fc7a12df95ca23f4d05fa" Oct 30 00:37:58.369991 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:37:58.367130 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf90f41004529a7c7f998958db88becac65c20dfc62fc7a12df95ca23f4d05fa\": container with ID starting with cf90f41004529a7c7f998958db88becac65c20dfc62fc7a12df95ca23f4d05fa not found: ID does not exist" containerID="cf90f41004529a7c7f998958db88becac65c20dfc62fc7a12df95ca23f4d05fa" Oct 30 00:37:58.369991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.367186 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf90f41004529a7c7f998958db88becac65c20dfc62fc7a12df95ca23f4d05fa"} err="failed to get container status \"cf90f41004529a7c7f998958db88becac65c20dfc62fc7a12df95ca23f4d05fa\": rpc error: code = NotFound desc = could not find container \"cf90f41004529a7c7f998958db88becac65c20dfc62fc7a12df95ca23f4d05fa\": container with ID starting with cf90f41004529a7c7f998958db88becac65c20dfc62fc7a12df95ca23f4d05fa not found: ID does not exist" Oct 30 00:37:58.369991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.367213 5001 scope.go:122] "RemoveContainer" containerID="fbe80941790a78584998bb0942f567da5bb31636353fd30b0cdcf7367a5595a0" Oct 30 00:37:58.380086 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.380062 5001 scope.go:122] "RemoveContainer" containerID="fbe80941790a78584998bb0942f567da5bb31636353fd30b0cdcf7367a5595a0" Oct 30 00:37:58.380402 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:37:58.380380 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbe80941790a78584998bb0942f567da5bb31636353fd30b0cdcf7367a5595a0\": container with ID starting with fbe80941790a78584998bb0942f567da5bb31636353fd30b0cdcf7367a5595a0 not found: ID does not exist" containerID="fbe80941790a78584998bb0942f567da5bb31636353fd30b0cdcf7367a5595a0" Oct 30 00:37:58.380449 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.380409 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbe80941790a78584998bb0942f567da5bb31636353fd30b0cdcf7367a5595a0"} err="failed to get container status \"fbe80941790a78584998bb0942f567da5bb31636353fd30b0cdcf7367a5595a0\": rpc error: code = NotFound desc = could not find container \"fbe80941790a78584998bb0942f567da5bb31636353fd30b0cdcf7367a5595a0\": container with ID starting with fbe80941790a78584998bb0942f567da5bb31636353fd30b0cdcf7367a5595a0 not found: ID does not exist" Oct 30 00:37:58.380449 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.380426 5001 scope.go:122] "RemoveContainer" containerID="6b697ac96104253924b98717b1255c234b39e296acada0ae409b63ea60d3159c" Oct 30 00:37:58.392044 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.392025 5001 scope.go:122] "RemoveContainer" containerID="6b697ac96104253924b98717b1255c234b39e296acada0ae409b63ea60d3159c" Oct 30 00:37:58.392314 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:37:58.392286 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b697ac96104253924b98717b1255c234b39e296acada0ae409b63ea60d3159c\": container with ID starting with 6b697ac96104253924b98717b1255c234b39e296acada0ae409b63ea60d3159c not found: ID does not exist" containerID="6b697ac96104253924b98717b1255c234b39e296acada0ae409b63ea60d3159c" Oct 30 00:37:58.392386 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.392311 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b697ac96104253924b98717b1255c234b39e296acada0ae409b63ea60d3159c"} err="failed to get container status \"6b697ac96104253924b98717b1255c234b39e296acada0ae409b63ea60d3159c\": rpc error: code = NotFound desc = could not find container \"6b697ac96104253924b98717b1255c234b39e296acada0ae409b63ea60d3159c\": container with ID starting with 6b697ac96104253924b98717b1255c234b39e296acada0ae409b63ea60d3159c not found: ID does not exist" Oct 30 00:37:58.392386 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.392328 5001 scope.go:122] "RemoveContainer" containerID="2424084982f40ddcac1405610a72c7eca9b7f6985742ea3921953c90f8649a98" Oct 30 00:37:58.404699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.404680 5001 scope.go:122] "RemoveContainer" containerID="2424084982f40ddcac1405610a72c7eca9b7f6985742ea3921953c90f8649a98" Oct 30 00:37:58.404997 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:37:58.404972 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2424084982f40ddcac1405610a72c7eca9b7f6985742ea3921953c90f8649a98\": container with ID starting with 2424084982f40ddcac1405610a72c7eca9b7f6985742ea3921953c90f8649a98 not found: ID does not exist" containerID="2424084982f40ddcac1405610a72c7eca9b7f6985742ea3921953c90f8649a98" Oct 30 00:37:58.405062 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.404998 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2424084982f40ddcac1405610a72c7eca9b7f6985742ea3921953c90f8649a98"} err="failed to get container status \"2424084982f40ddcac1405610a72c7eca9b7f6985742ea3921953c90f8649a98\": rpc error: code = NotFound desc = could not find container \"2424084982f40ddcac1405610a72c7eca9b7f6985742ea3921953c90f8649a98\": container with ID starting with 2424084982f40ddcac1405610a72c7eca9b7f6985742ea3921953c90f8649a98 not found: ID does not exist" Oct 30 00:37:58.463508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.463460 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx"] Oct 30 00:37:58.483030 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.481150 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54658\" (UniqueName: \"kubernetes.io/projected/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-kube-api-access-54658\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.483030 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.481192 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.483030 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.481209 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.483030 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.481223 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.483030 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.481236 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a414d8-654e-4a80-bfaf-1ea96f68ea8e-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:37:58.494579 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.494526 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-rpvsx"] Oct 30 00:37:58.557834 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.557804 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687"] Oct 30 00:37:58.567625 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.567594 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-8v687"] Oct 30 00:37:58.572820 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.572788 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf"] Oct 30 00:37:58.578494 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.578451 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-c66gf"] Oct 30 00:37:58.681213 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.681182 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn"] Oct 30 00:37:58.687099 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.687074 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a7ac6-openshift-default-798b6bc7df-ntfxn"] Oct 30 00:37:58.920956 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.920848 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74a414d8-654e-4a80-bfaf-1ea96f68ea8e" path="/var/lib/kubelet/pods/74a414d8-654e-4a80-bfaf-1ea96f68ea8e/volumes" Oct 30 00:37:58.921441 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.921424 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad5882d9-90e8-400f-90d5-d4ba5d8da5fc" path="/var/lib/kubelet/pods/ad5882d9-90e8-400f-90d5-d4ba5d8da5fc/volumes" Oct 30 00:37:58.921839 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.921826 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8e98d30-8619-44a3-81eb-8d2a2e7c2af5" path="/var/lib/kubelet/pods/b8e98d30-8619-44a3-81eb-8d2a2e7c2af5/volumes" Oct 30 00:37:58.922607 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.922591 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d34b94b7-c280-4d34-a4b3-5ae295c595cb" path="/var/lib/kubelet/pods/d34b94b7-c280-4d34-a4b3-5ae295c595cb/volumes" Oct 30 00:37:58.923076 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:58.923062 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea34fb77-6b08-4939-b1c0-9ba33e8afd0f" path="/var/lib/kubelet/pods/ea34fb77-6b08-4939-b1c0-9ba33e8afd0f/volumes" Oct 30 00:37:59.152562 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.152525 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-545e045c-f9cf7"] Oct 30 00:37:59.153198 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.153183 5001 cpu_manager.go:414] "containerMap: removing container" podUID="a76339ab-86f8-4e69-8844-730227c58fd7" containerName="revoke-keys" Oct 30 00:37:59.153250 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.153199 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a76339ab-86f8-4e69-8844-730227c58fd7" containerName="revoke-keys" Oct 30 00:37:59.153250 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.153215 5001 cpu_manager.go:414] "containerMap: removing container" podUID="ad5882d9-90e8-400f-90d5-d4ba5d8da5fc" containerName="istio-proxy" Oct 30 00:37:59.153250 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.153220 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad5882d9-90e8-400f-90d5-d4ba5d8da5fc" containerName="istio-proxy" Oct 30 00:37:59.153250 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.153231 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d34b94b7-c280-4d34-a4b3-5ae295c595cb" containerName="istio-proxy" Oct 30 00:37:59.153250 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.153237 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d34b94b7-c280-4d34-a4b3-5ae295c595cb" containerName="istio-proxy" Oct 30 00:37:59.153250 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.153250 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4deafe88-91e1-4878-9203-e28de81ce261" containerName="istio-proxy" Oct 30 00:37:59.153446 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.153256 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4deafe88-91e1-4878-9203-e28de81ce261" containerName="istio-proxy" Oct 30 00:37:59.153446 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.153265 5001 cpu_manager.go:414] "containerMap: removing container" podUID="ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a" containerName="revoke-keys" Oct 30 00:37:59.153446 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.153270 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a" containerName="revoke-keys" Oct 30 00:37:59.153446 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.153277 5001 cpu_manager.go:414] "containerMap: removing container" podUID="74a414d8-654e-4a80-bfaf-1ea96f68ea8e" containerName="istio-proxy" Oct 30 00:37:59.153446 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.153282 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74a414d8-654e-4a80-bfaf-1ea96f68ea8e" containerName="istio-proxy" Oct 30 00:37:59.153446 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.153290 5001 cpu_manager.go:414] "containerMap: removing container" podUID="ea34fb77-6b08-4939-b1c0-9ba33e8afd0f" containerName="istio-proxy" Oct 30 00:37:59.153446 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.153294 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea34fb77-6b08-4939-b1c0-9ba33e8afd0f" containerName="istio-proxy" Oct 30 00:37:59.153446 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.153308 5001 cpu_manager.go:414] "containerMap: removing container" podUID="92fb1547-83d7-4b71-884d-b9fde3d388ec" containerName="revoke-keys" Oct 30 00:37:59.153446 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.153312 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92fb1547-83d7-4b71-884d-b9fde3d388ec" containerName="revoke-keys" Oct 30 00:37:59.153446 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.153324 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4deafe88-91e1-4878-9203-e28de81ce261" containerName="istio-proxy" Oct 30 00:37:59.153446 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.153328 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4deafe88-91e1-4878-9203-e28de81ce261" containerName="istio-proxy" Oct 30 00:37:59.153446 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.153411 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d34b94b7-c280-4d34-a4b3-5ae295c595cb" containerName="istio-proxy" Oct 30 00:37:59.153446 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.153421 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4deafe88-91e1-4878-9203-e28de81ce261" containerName="istio-proxy" Oct 30 00:37:59.153446 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.153429 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a" containerName="revoke-keys" Oct 30 00:37:59.153446 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.153434 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="74a414d8-654e-4a80-bfaf-1ea96f68ea8e" containerName="istio-proxy" Oct 30 00:37:59.153446 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.153439 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea34fb77-6b08-4939-b1c0-9ba33e8afd0f" containerName="istio-proxy" Oct 30 00:37:59.153446 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.153447 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="92fb1547-83d7-4b71-884d-b9fde3d388ec" containerName="revoke-keys" Oct 30 00:37:59.154050 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.153457 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a76339ab-86f8-4e69-8844-730227c58fd7" containerName="revoke-keys" Oct 30 00:37:59.154050 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.153464 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad5882d9-90e8-400f-90d5-d4ba5d8da5fc" containerName="istio-proxy" Oct 30 00:37:59.158830 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.158807 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-545e045c-f9cf7" Oct 30 00:37:59.171896 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.171828 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:37:59.171896 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.171862 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:37:59.172096 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.172039 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:37:59.172096 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.172084 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:37:59.177746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.177721 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-545e045c-f9cf7"] Oct 30 00:37:59.291742 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.291711 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd2803cf-8270-46c9-92ae-762022d4e340-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-545e045c-f9cf7\" (UID: \"cd2803cf-8270-46c9-92ae-762022d4e340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-545e045c-f9cf7" Oct 30 00:37:59.291955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.291754 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd2803cf-8270-46c9-92ae-762022d4e340-maas-api-xks-ca\") pod \"maas-api-revoke-sub-545e045c-f9cf7\" (UID: \"cd2803cf-8270-46c9-92ae-762022d4e340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-545e045c-f9cf7" Oct 30 00:37:59.292028 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.291989 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbpvx\" (UniqueName: \"kubernetes.io/projected/cd2803cf-8270-46c9-92ae-762022d4e340-kube-api-access-qbpvx\") pod \"maas-api-revoke-sub-545e045c-f9cf7\" (UID: \"cd2803cf-8270-46c9-92ae-762022d4e340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-545e045c-f9cf7" Oct 30 00:37:59.393170 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.393130 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd2803cf-8270-46c9-92ae-762022d4e340-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-545e045c-f9cf7\" (UID: \"cd2803cf-8270-46c9-92ae-762022d4e340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-545e045c-f9cf7" Oct 30 00:37:59.393348 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.393209 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd2803cf-8270-46c9-92ae-762022d4e340-maas-api-xks-ca\") pod \"maas-api-revoke-sub-545e045c-f9cf7\" (UID: \"cd2803cf-8270-46c9-92ae-762022d4e340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-545e045c-f9cf7" Oct 30 00:37:59.393450 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.393435 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbpvx\" (UniqueName: \"kubernetes.io/projected/cd2803cf-8270-46c9-92ae-762022d4e340-kube-api-access-qbpvx\") pod \"maas-api-revoke-sub-545e045c-f9cf7\" (UID: \"cd2803cf-8270-46c9-92ae-762022d4e340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-545e045c-f9cf7" Oct 30 00:37:59.393927 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.393889 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd2803cf-8270-46c9-92ae-762022d4e340-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-545e045c-f9cf7\" (UID: \"cd2803cf-8270-46c9-92ae-762022d4e340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-545e045c-f9cf7" Oct 30 00:37:59.395772 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.395752 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd2803cf-8270-46c9-92ae-762022d4e340-maas-api-xks-ca\") pod \"maas-api-revoke-sub-545e045c-f9cf7\" (UID: \"cd2803cf-8270-46c9-92ae-762022d4e340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-545e045c-f9cf7" Oct 30 00:37:59.414266 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.414237 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbpvx\" (UniqueName: \"kubernetes.io/projected/cd2803cf-8270-46c9-92ae-762022d4e340-kube-api-access-qbpvx\") pod \"maas-api-revoke-sub-545e045c-f9cf7\" (UID: \"cd2803cf-8270-46c9-92ae-762022d4e340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-545e045c-f9cf7" Oct 30 00:37:59.474248 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.474176 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-545e045c-f9cf7" Oct 30 00:37:59.783267 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:37:59.783238 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd2803cf_8270_46c9_92ae_762022d4e340.slice/crio-310a58cc9ac71f35c2289f32dda4d893405b7ef076d5ef2ccf18bd540d1f45d5 WatchSource:0}: Error finding container 310a58cc9ac71f35c2289f32dda4d893405b7ef076d5ef2ccf18bd540d1f45d5: Status 404 returned error can't find the container with id 310a58cc9ac71f35c2289f32dda4d893405b7ef076d5ef2ccf18bd540d1f45d5 Oct 30 00:37:59.783384 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.783281 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-545e045c-f9cf7"] Oct 30 00:37:59.991709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:37:59.991677 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-29f3ca-6765676c44-vvgqp"] Oct 30 00:38:00.389772 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:00.389679 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-545e045c-f9cf7" event={"ID":"cd2803cf-8270-46c9-92ae-762022d4e340","Type":"ContainerStarted","Data":"87875f2cd0e67444ffefafed597a3bac8e69cc7d0fce511a7067738e8fe80e44"} Oct 30 00:38:00.389772 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:00.389741 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-545e045c-f9cf7" event={"ID":"cd2803cf-8270-46c9-92ae-762022d4e340","Type":"ContainerStarted","Data":"310a58cc9ac71f35c2289f32dda4d893405b7ef076d5ef2ccf18bd540d1f45d5"} Oct 30 00:38:00.427583 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:00.427532 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-545e045c-f9cf7" podStartSLOduration=1.427517636 podStartE2EDuration="1.427517636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:37:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:00.424302252 +0000 UTC m=+1995.629803469" watchObservedRunningTime="2026-10-30 00:38:00.427517636 +0000 UTC m=+1995.633018868" Oct 30 00:38:01.052303 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:01.052260 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fad7d195-9gjm9"] Oct 30 00:38:01.082933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:01.078759 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fad7d195-9gjm9"] Oct 30 00:38:01.227720 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:01.227689 5001 scope.go:122] "RemoveContainer" containerID="9427d368df6f5ffd1d0a2f8be9f4336d79060fb44f7baeedb7259ffc70e4ee89" Oct 30 00:38:01.239178 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:01.239158 5001 scope.go:122] "RemoveContainer" containerID="4fc8ab6c8a3e9e175412fc0d2b99aead7879429f71ba3a90773aba9d3d7abe58" Oct 30 00:38:01.249385 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:01.249359 5001 scope.go:122] "RemoveContainer" containerID="2460abf827d74394c9b7e3a6c689d116c9d7f324513050df272e29ca8a6d577c" Oct 30 00:38:01.259327 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:01.259309 5001 scope.go:122] "RemoveContainer" containerID="d3692f4a0d7e9fc815f4a2d7e41f1ee6b9df3d6d6f30a8671ef5c4b56cb5a6fc" Oct 30 00:38:01.269682 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:01.269655 5001 scope.go:122] "RemoveContainer" containerID="d6f233579d5c6b19fcd5a7b398fc86f945e638ca70618baf7fed77261f565c15" Oct 30 00:38:01.279743 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:01.279724 5001 scope.go:122] "RemoveContainer" containerID="29c030eac9a6780fbf2e0ce66af394545f18630b614c59a925ff9b03e464a9f3" Oct 30 00:38:01.400336 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:01.400301 5001 generic.go:356] "Generic (PLEG): container finished" podID="cd2803cf-8270-46c9-92ae-762022d4e340" containerID="87875f2cd0e67444ffefafed597a3bac8e69cc7d0fce511a7067738e8fe80e44" exitCode=0 Oct 30 00:38:01.400706 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:01.400341 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-545e045c-f9cf7" event={"ID":"cd2803cf-8270-46c9-92ae-762022d4e340","Type":"ContainerDied","Data":"87875f2cd0e67444ffefafed597a3bac8e69cc7d0fce511a7067738e8fe80e44"} Oct 30 00:38:01.850850 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:01.850808 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c1eeb6-7d97d86c79-zmp4b" podUID="2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:38:02.585270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:02.585240 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-545e045c-f9cf7" Oct 30 00:38:02.745156 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:02.745048 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd2803cf-8270-46c9-92ae-762022d4e340-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd2803cf-8270-46c9-92ae-762022d4e340-maas-api-xks-ca\") pod \"cd2803cf-8270-46c9-92ae-762022d4e340\" (UID: \"cd2803cf-8270-46c9-92ae-762022d4e340\") " Oct 30 00:38:02.745156 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:02.745111 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd2803cf-8270-46c9-92ae-762022d4e340-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd2803cf-8270-46c9-92ae-762022d4e340-maas-api-openshift-ca\") pod \"cd2803cf-8270-46c9-92ae-762022d4e340\" (UID: \"cd2803cf-8270-46c9-92ae-762022d4e340\") " Oct 30 00:38:02.745396 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:02.745378 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd2803cf-8270-46c9-92ae-762022d4e340-kube-api-access-qbpvx\" (UniqueName: \"kubernetes.io/projected/cd2803cf-8270-46c9-92ae-762022d4e340-kube-api-access-qbpvx\") pod \"cd2803cf-8270-46c9-92ae-762022d4e340\" (UID: \"cd2803cf-8270-46c9-92ae-762022d4e340\") " Oct 30 00:38:02.745575 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:02.745546 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd2803cf-8270-46c9-92ae-762022d4e340-maas-api-openshift-ca" pod "cd2803cf-8270-46c9-92ae-762022d4e340" (UID: "cd2803cf-8270-46c9-92ae-762022d4e340"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:38:02.745960 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:02.745903 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd2803cf-8270-46c9-92ae-762022d4e340-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:02.747664 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:02.747636 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd2803cf-8270-46c9-92ae-762022d4e340-kube-api-access-qbpvx" pod "cd2803cf-8270-46c9-92ae-762022d4e340" (UID: "cd2803cf-8270-46c9-92ae-762022d4e340"). InnerVolumeSpecName "kube-api-access-qbpvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:02.751036 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:02.751014 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd2803cf-8270-46c9-92ae-762022d4e340-maas-api-xks-ca" pod "cd2803cf-8270-46c9-92ae-762022d4e340" (UID: "cd2803cf-8270-46c9-92ae-762022d4e340"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:38:02.785690 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:02.785652 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ecb34a-7c4f79cf86-6l8nz" podUID="0261228f-3733-4993-893e-43c742c7bd5e" containerName="payload-processing" containerID="cri-o://f333c84868e04ee11a4d80dc0251fa3008a4c0ca379f0ce0a97a7166ba0a9e2a" gracePeriod=25 Oct 30 00:38:02.805206 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:02.805157 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ecb34a-7ffdbf96c-8lgp2" podUID="aca30d4b-8350-4bfd-ba90-54e70dacf765" containerName="payload-pre-processing" containerID="cri-o://2d9c1cf0fefda535b52b87921c9e3752803c6e1de1a90bc46be95d007cd32f38" gracePeriod=25 Oct 30 00:38:02.847277 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:02.847238 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbpvx\" (UniqueName: \"kubernetes.io/projected/cd2803cf-8270-46c9-92ae-762022d4e340-kube-api-access-qbpvx\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:02.847277 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:02.847270 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd2803cf-8270-46c9-92ae-762022d4e340-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:02.924939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:02.923716 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca471334-1301-43ac-886e-90d3610b5ea9" path="/var/lib/kubelet/pods/ca471334-1301-43ac-886e-90d3610b5ea9/volumes" Oct 30 00:38:03.374900 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:03.374870 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ecb34a-d62vd"] Oct 30 00:38:03.387084 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:03.387054 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ecb34a-d62vd"] Oct 30 00:38:03.430781 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:03.428081 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-545e045c-f9cf7" Oct 30 00:38:03.430781 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:03.428129 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-545e045c-f9cf7" event={"ID":"cd2803cf-8270-46c9-92ae-762022d4e340","Type":"ContainerDied","Data":"310a58cc9ac71f35c2289f32dda4d893405b7ef076d5ef2ccf18bd540d1f45d5"} Oct 30 00:38:03.430781 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:03.428190 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="310a58cc9ac71f35c2289f32dda4d893405b7ef076d5ef2ccf18bd540d1f45d5" Oct 30 00:38:03.431113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:03.431027 5001 generic.go:356] "Generic (PLEG): container finished" podID="2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c" containerID="83e85ac636e6cbf91083e1187a5c2965a1a6265f3c27a4ecde281ec1daec8ecb" exitCode=0 Oct 30 00:38:03.431210 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:03.431186 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c1eeb6-7d97d86c79-zmp4b" event={"ID":"2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c","Type":"ContainerDied","Data":"83e85ac636e6cbf91083e1187a5c2965a1a6265f3c27a4ecde281ec1daec8ecb"} Oct 30 00:38:03.533167 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:03.533144 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c1eeb6-7d97d86c79-zmp4b" Oct 30 00:38:03.673718 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:03.673687 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c-plugins-config-volume\") pod \"2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c\" (UID: \"2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c\") " Oct 30 00:38:03.674227 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:03.673734 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c-kube-api-access-ks56w\" (UniqueName: \"kubernetes.io/projected/2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c-kube-api-access-ks56w\") pod \"2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c\" (UID: \"2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c\") " Oct 30 00:38:03.674227 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:03.674147 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c-plugins-config-volume" pod "2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c" (UID: "2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:38:03.674496 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:03.674477 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:03.675979 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:03.675954 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c-kube-api-access-ks56w" pod "2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c" (UID: "2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c"). InnerVolumeSpecName "kube-api-access-ks56w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:03.776236 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:03.776178 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks56w\" (UniqueName: \"kubernetes.io/projected/2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c-kube-api-access-ks56w\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:03.889768 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:03.889724 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl" podUID="1be5133e-f463-4b8a-bc62-cd8a863ec1c7" containerName="main" containerID="cri-o://5ba3be446dee52c498e35d0691d39c9bbcc11a0410299efedaa4eb4e5fbb36bb" gracePeriod=45 Oct 30 00:38:04.176690 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.176663 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl" Oct 30 00:38:04.282042 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.282011 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-home\" (UniqueName: \"kubernetes.io/empty-dir/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-home\") pod \"1be5133e-f463-4b8a-bc62-cd8a863ec1c7\" (UID: \"1be5133e-f463-4b8a-bc62-cd8a863ec1c7\") " Oct 30 00:38:04.282217 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.282066 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-model-cache\") pod \"1be5133e-f463-4b8a-bc62-cd8a863ec1c7\" (UID: \"1be5133e-f463-4b8a-bc62-cd8a863ec1c7\") " Oct 30 00:38:04.282217 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.282172 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-tls-certs\" (UniqueName: \"kubernetes.io/secret/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-tls-certs\") pod \"1be5133e-f463-4b8a-bc62-cd8a863ec1c7\" (UID: \"1be5133e-f463-4b8a-bc62-cd8a863ec1c7\") " Oct 30 00:38:04.282302 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.282272 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-tmp-dir\") pod \"1be5133e-f463-4b8a-bc62-cd8a863ec1c7\" (UID: \"1be5133e-f463-4b8a-bc62-cd8a863ec1c7\") " Oct 30 00:38:04.282346 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.282287 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-home" pod "1be5133e-f463-4b8a-bc62-cd8a863ec1c7" (UID: "1be5133e-f463-4b8a-bc62-cd8a863ec1c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:04.282346 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.282317 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-kube-api-access-vnpg5\" (UniqueName: \"kubernetes.io/projected/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-kube-api-access-vnpg5\") pod \"1be5133e-f463-4b8a-bc62-cd8a863ec1c7\" (UID: \"1be5133e-f463-4b8a-bc62-cd8a863ec1c7\") " Oct 30 00:38:04.282428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.282367 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-model-cache" pod "1be5133e-f463-4b8a-bc62-cd8a863ec1c7" (UID: "1be5133e-f463-4b8a-bc62-cd8a863ec1c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:04.282428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.282375 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-dshm\") pod \"1be5133e-f463-4b8a-bc62-cd8a863ec1c7\" (UID: \"1be5133e-f463-4b8a-bc62-cd8a863ec1c7\") " Oct 30 00:38:04.282586 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.282555 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-tmp-dir" pod "1be5133e-f463-4b8a-bc62-cd8a863ec1c7" (UID: "1be5133e-f463-4b8a-bc62-cd8a863ec1c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:04.282893 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.282868 5001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-tmp-dir\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:04.282893 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.282890 5001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-home\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:04.282893 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.282899 5001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-model-cache\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:04.284517 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.284495 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-dshm" pod "1be5133e-f463-4b8a-bc62-cd8a863ec1c7" (UID: "1be5133e-f463-4b8a-bc62-cd8a863ec1c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:04.284638 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.284518 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-tls-certs" pod "1be5133e-f463-4b8a-bc62-cd8a863ec1c7" (UID: "1be5133e-f463-4b8a-bc62-cd8a863ec1c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:38:04.284638 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.284595 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-kube-api-access-vnpg5" pod "1be5133e-f463-4b8a-bc62-cd8a863ec1c7" (UID: "1be5133e-f463-4b8a-bc62-cd8a863ec1c7"). InnerVolumeSpecName "kube-api-access-vnpg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:04.339609 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.339499 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn" podUID="cdd8c42e-35c2-4141-afab-e400b23a9163" containerName="main" containerID="cri-o://63e5202d8c75d8622d61709ccbcd213eb2a8777a9567cf02b520968278bb99c2" gracePeriod=45 Oct 30 00:38:04.384116 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.384083 5001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-tls-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:04.384242 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.384118 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnpg5\" (UniqueName: \"kubernetes.io/projected/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-kube-api-access-vnpg5\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:04.384242 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.384135 5001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1be5133e-f463-4b8a-bc62-cd8a863ec1c7-dshm\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:04.440726 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.440686 5001 generic.go:356] "Generic (PLEG): container finished" podID="1be5133e-f463-4b8a-bc62-cd8a863ec1c7" containerID="5ba3be446dee52c498e35d0691d39c9bbcc11a0410299efedaa4eb4e5fbb36bb" exitCode=0 Oct 30 00:38:04.440868 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.440799 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl" Oct 30 00:38:04.440868 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.440815 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl" event={"ID":"1be5133e-f463-4b8a-bc62-cd8a863ec1c7","Type":"ContainerDied","Data":"5ba3be446dee52c498e35d0691d39c9bbcc11a0410299efedaa4eb4e5fbb36bb"} Oct 30 00:38:04.440868 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.440859 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl" event={"ID":"1be5133e-f463-4b8a-bc62-cd8a863ec1c7","Type":"ContainerDied","Data":"9be7bb56d9d9fc97866bf3a370ef5283e82415a18ea790ed7a68b2c267578a3e"} Oct 30 00:38:04.441070 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.440878 5001 scope.go:122] "RemoveContainer" containerID="5ba3be446dee52c498e35d0691d39c9bbcc11a0410299efedaa4eb4e5fbb36bb" Oct 30 00:38:04.443417 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.442960 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c1eeb6-7d97d86c79-zmp4b" Oct 30 00:38:04.443417 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.442962 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c1eeb6-7d97d86c79-zmp4b" event={"ID":"2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c","Type":"ContainerDied","Data":"61e3eb4047d735c4a9d1f18f64ae125c3313dab2cdfbb4a13ced0c8300256adb"} Oct 30 00:38:04.455648 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.455633 5001 scope.go:122] "RemoveContainer" containerID="5ba3be446dee52c498e35d0691d39c9bbcc11a0410299efedaa4eb4e5fbb36bb" Oct 30 00:38:04.455975 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:38:04.455952 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ba3be446dee52c498e35d0691d39c9bbcc11a0410299efedaa4eb4e5fbb36bb\": container with ID starting with 5ba3be446dee52c498e35d0691d39c9bbcc11a0410299efedaa4eb4e5fbb36bb not found: ID does not exist" containerID="5ba3be446dee52c498e35d0691d39c9bbcc11a0410299efedaa4eb4e5fbb36bb" Oct 30 00:38:04.456042 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.455986 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ba3be446dee52c498e35d0691d39c9bbcc11a0410299efedaa4eb4e5fbb36bb"} err="failed to get container status \"5ba3be446dee52c498e35d0691d39c9bbcc11a0410299efedaa4eb4e5fbb36bb\": rpc error: code = NotFound desc = could not find container \"5ba3be446dee52c498e35d0691d39c9bbcc11a0410299efedaa4eb4e5fbb36bb\": container with ID starting with 5ba3be446dee52c498e35d0691d39c9bbcc11a0410299efedaa4eb4e5fbb36bb not found: ID does not exist" Oct 30 00:38:04.456042 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.456009 5001 scope.go:122] "RemoveContainer" containerID="83e85ac636e6cbf91083e1187a5c2965a1a6265f3c27a4ecde281ec1daec8ecb" Oct 30 00:38:04.487540 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.487506 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl"] Oct 30 00:38:04.496610 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.496580 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a782e5/auth-test-model-a782e5-kserve-7596bffcbf-m68sl"] Oct 30 00:38:04.553957 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.549976 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c1eeb6-7d97d86c79-zmp4b"] Oct 30 00:38:04.553957 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.553166 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c1eeb6-7d97d86c79-zmp4b"] Oct 30 00:38:04.616866 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.616840 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn" Oct 30 00:38:04.789562 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.789532 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdd8c42e-35c2-4141-afab-e400b23a9163-tls-certs\" (UniqueName: \"kubernetes.io/secret/cdd8c42e-35c2-4141-afab-e400b23a9163-tls-certs\") pod \"cdd8c42e-35c2-4141-afab-e400b23a9163\" (UID: \"cdd8c42e-35c2-4141-afab-e400b23a9163\") " Oct 30 00:38:04.790146 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.789571 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdd8c42e-35c2-4141-afab-e400b23a9163-home\" (UniqueName: \"kubernetes.io/empty-dir/cdd8c42e-35c2-4141-afab-e400b23a9163-home\") pod \"cdd8c42e-35c2-4141-afab-e400b23a9163\" (UID: \"cdd8c42e-35c2-4141-afab-e400b23a9163\") " Oct 30 00:38:04.790146 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.789698 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdd8c42e-35c2-4141-afab-e400b23a9163-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdd8c42e-35c2-4141-afab-e400b23a9163-dshm\") pod \"cdd8c42e-35c2-4141-afab-e400b23a9163\" (UID: \"cdd8c42e-35c2-4141-afab-e400b23a9163\") " Oct 30 00:38:04.790146 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.789757 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdd8c42e-35c2-4141-afab-e400b23a9163-kube-api-access-f4mf2\" (UniqueName: \"kubernetes.io/projected/cdd8c42e-35c2-4141-afab-e400b23a9163-kube-api-access-f4mf2\") pod \"cdd8c42e-35c2-4141-afab-e400b23a9163\" (UID: \"cdd8c42e-35c2-4141-afab-e400b23a9163\") " Oct 30 00:38:04.790146 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.789880 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd8c42e-35c2-4141-afab-e400b23a9163-home" pod "cdd8c42e-35c2-4141-afab-e400b23a9163" (UID: "cdd8c42e-35c2-4141-afab-e400b23a9163"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:04.790146 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.789891 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdd8c42e-35c2-4141-afab-e400b23a9163-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdd8c42e-35c2-4141-afab-e400b23a9163-tmp-dir\") pod \"cdd8c42e-35c2-4141-afab-e400b23a9163\" (UID: \"cdd8c42e-35c2-4141-afab-e400b23a9163\") " Oct 30 00:38:04.790146 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.790027 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdd8c42e-35c2-4141-afab-e400b23a9163-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdd8c42e-35c2-4141-afab-e400b23a9163-model-cache\") pod \"cdd8c42e-35c2-4141-afab-e400b23a9163\" (UID: \"cdd8c42e-35c2-4141-afab-e400b23a9163\") " Oct 30 00:38:04.790146 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.790134 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd8c42e-35c2-4141-afab-e400b23a9163-tmp-dir" pod "cdd8c42e-35c2-4141-afab-e400b23a9163" (UID: "cdd8c42e-35c2-4141-afab-e400b23a9163"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:04.790546 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.790271 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd8c42e-35c2-4141-afab-e400b23a9163-model-cache" pod "cdd8c42e-35c2-4141-afab-e400b23a9163" (UID: "cdd8c42e-35c2-4141-afab-e400b23a9163"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:04.790618 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.790584 5001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdd8c42e-35c2-4141-afab-e400b23a9163-home\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:04.790618 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.790600 5001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdd8c42e-35c2-4141-afab-e400b23a9163-tmp-dir\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:04.790618 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.790609 5001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdd8c42e-35c2-4141-afab-e400b23a9163-model-cache\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:04.792104 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.792080 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd8c42e-35c2-4141-afab-e400b23a9163-dshm" pod "cdd8c42e-35c2-4141-afab-e400b23a9163" (UID: "cdd8c42e-35c2-4141-afab-e400b23a9163"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:04.792318 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.792295 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd8c42e-35c2-4141-afab-e400b23a9163-kube-api-access-f4mf2" pod "cdd8c42e-35c2-4141-afab-e400b23a9163" (UID: "cdd8c42e-35c2-4141-afab-e400b23a9163"). InnerVolumeSpecName "kube-api-access-f4mf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:04.792365 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.792303 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdd8c42e-35c2-4141-afab-e400b23a9163-tls-certs" pod "cdd8c42e-35c2-4141-afab-e400b23a9163" (UID: "cdd8c42e-35c2-4141-afab-e400b23a9163"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:38:04.891939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.891837 5001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdd8c42e-35c2-4141-afab-e400b23a9163-tls-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:04.891939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.891863 5001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdd8c42e-35c2-4141-afab-e400b23a9163-dshm\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:04.891939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.891873 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4mf2\" (UniqueName: \"kubernetes.io/projected/cdd8c42e-35c2-4141-afab-e400b23a9163-kube-api-access-f4mf2\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:04.920715 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.920690 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1be5133e-f463-4b8a-bc62-cd8a863ec1c7" path="/var/lib/kubelet/pods/1be5133e-f463-4b8a-bc62-cd8a863ec1c7/volumes" Oct 30 00:38:04.921149 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.921134 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c" path="/var/lib/kubelet/pods/2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c/volumes" Oct 30 00:38:04.921553 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.921540 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92fb1547-83d7-4b71-884d-b9fde3d388ec" path="/var/lib/kubelet/pods/92fb1547-83d7-4b71-884d-b9fde3d388ec/volumes" Oct 30 00:38:04.991977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:04.991905 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-29f3ca-6765676c44-vvgqp" podUID="d4c8106e-3826-4f4d-b267-93ecea650c90" containerName="payload-processing" containerID="cri-o://5145862a61b8d6ac9966ff9c40475f0ae6d733ccdc3404f250ce8b1617fe5821" gracePeriod=25 Oct 30 00:38:05.456256 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:05.456219 5001 generic.go:356] "Generic (PLEG): container finished" podID="cdd8c42e-35c2-4141-afab-e400b23a9163" containerID="63e5202d8c75d8622d61709ccbcd213eb2a8777a9567cf02b520968278bb99c2" exitCode=0 Oct 30 00:38:05.456256 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:05.456259 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn" Oct 30 00:38:05.456480 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:05.456304 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn" event={"ID":"cdd8c42e-35c2-4141-afab-e400b23a9163","Type":"ContainerDied","Data":"63e5202d8c75d8622d61709ccbcd213eb2a8777a9567cf02b520968278bb99c2"} Oct 30 00:38:05.456480 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:05.456343 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn" event={"ID":"cdd8c42e-35c2-4141-afab-e400b23a9163","Type":"ContainerDied","Data":"c4da90fddd593ef94872f459347146658a09b1151b403a49dd80b0ec5b14ecee"} Oct 30 00:38:05.456480 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:05.456361 5001 scope.go:122] "RemoveContainer" containerID="63e5202d8c75d8622d61709ccbcd213eb2a8777a9567cf02b520968278bb99c2" Oct 30 00:38:05.468029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:05.468012 5001 scope.go:122] "RemoveContainer" containerID="63e5202d8c75d8622d61709ccbcd213eb2a8777a9567cf02b520968278bb99c2" Oct 30 00:38:05.468345 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:38:05.468326 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63e5202d8c75d8622d61709ccbcd213eb2a8777a9567cf02b520968278bb99c2\": container with ID starting with 63e5202d8c75d8622d61709ccbcd213eb2a8777a9567cf02b520968278bb99c2 not found: ID does not exist" containerID="63e5202d8c75d8622d61709ccbcd213eb2a8777a9567cf02b520968278bb99c2" Oct 30 00:38:05.468414 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:05.468353 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63e5202d8c75d8622d61709ccbcd213eb2a8777a9567cf02b520968278bb99c2"} err="failed to get container status \"63e5202d8c75d8622d61709ccbcd213eb2a8777a9567cf02b520968278bb99c2\": rpc error: code = NotFound desc = could not find container \"63e5202d8c75d8622d61709ccbcd213eb2a8777a9567cf02b520968278bb99c2\": container with ID starting with 63e5202d8c75d8622d61709ccbcd213eb2a8777a9567cf02b520968278bb99c2 not found: ID does not exist" Oct 30 00:38:05.502127 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:05.502094 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn"] Oct 30 00:38:05.511888 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:05.511859 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-415d24/auth-test-model-415d24-kserve-cd4d594f7-d6ksn"] Oct 30 00:38:06.041795 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.040794 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3e94c5f-hr2vp"] Oct 30 00:38:06.048832 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.048768 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3e94c5f-hr2vp"] Oct 30 00:38:06.056168 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.056130 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz"] Oct 30 00:38:06.352020 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.351923 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a782e5-89zws"] Oct 30 00:38:06.352555 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.352541 5001 cpu_manager.go:414] "containerMap: removing container" podUID="cd2803cf-8270-46c9-92ae-762022d4e340" containerName="revoke-keys" Oct 30 00:38:06.352596 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.352557 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd2803cf-8270-46c9-92ae-762022d4e340" containerName="revoke-keys" Oct 30 00:38:06.352596 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.352575 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1be5133e-f463-4b8a-bc62-cd8a863ec1c7" containerName="main" Oct 30 00:38:06.352596 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.352580 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1be5133e-f463-4b8a-bc62-cd8a863ec1c7" containerName="main" Oct 30 00:38:06.352714 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.352596 5001 cpu_manager.go:414] "containerMap: removing container" podUID="cdd8c42e-35c2-4141-afab-e400b23a9163" containerName="main" Oct 30 00:38:06.352714 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.352602 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdd8c42e-35c2-4141-afab-e400b23a9163" containerName="main" Oct 30 00:38:06.352714 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.352615 5001 cpu_manager.go:414] "containerMap: removing container" podUID="2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c" containerName="payload-pre-processing" Oct 30 00:38:06.352714 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.352620 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c" containerName="payload-pre-processing" Oct 30 00:38:06.352841 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.352718 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd2803cf-8270-46c9-92ae-762022d4e340" containerName="revoke-keys" Oct 30 00:38:06.352841 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.352726 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4deafe88-91e1-4878-9203-e28de81ce261" containerName="istio-proxy" Oct 30 00:38:06.352841 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.352735 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1be5133e-f463-4b8a-bc62-cd8a863ec1c7" containerName="main" Oct 30 00:38:06.352841 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.352742 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdd8c42e-35c2-4141-afab-e400b23a9163" containerName="main" Oct 30 00:38:06.352841 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.352748 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c4937e3-cebc-42a4-adc3-2d72f6aa2f5c" containerName="payload-pre-processing" Oct 30 00:38:06.357852 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.357832 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a782e5-89zws" Oct 30 00:38:06.371470 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.371441 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:38:06.371614 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.371559 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:38:06.377075 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.377050 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a782e5-89zws"] Oct 30 00:38:06.406875 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.406838 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:38:06.407050 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.406961 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:38:06.516816 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.516781 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8q2g\" (UniqueName: \"kubernetes.io/projected/a0c236a4-aee1-407f-85a3-49f13a1f0d0b-kube-api-access-g8q2g\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a782e5-89zws\" (UID: \"a0c236a4-aee1-407f-85a3-49f13a1f0d0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a782e5-89zws" Oct 30 00:38:06.517043 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.517010 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c236a4-aee1-407f-85a3-49f13a1f0d0b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a782e5-89zws\" (UID: \"a0c236a4-aee1-407f-85a3-49f13a1f0d0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a782e5-89zws" Oct 30 00:38:06.517095 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.517065 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c236a4-aee1-407f-85a3-49f13a1f0d0b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a782e5-89zws\" (UID: \"a0c236a4-aee1-407f-85a3-49f13a1f0d0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a782e5-89zws" Oct 30 00:38:06.618479 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.618389 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8q2g\" (UniqueName: \"kubernetes.io/projected/a0c236a4-aee1-407f-85a3-49f13a1f0d0b-kube-api-access-g8q2g\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a782e5-89zws\" (UID: \"a0c236a4-aee1-407f-85a3-49f13a1f0d0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a782e5-89zws" Oct 30 00:38:06.618655 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.618601 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c236a4-aee1-407f-85a3-49f13a1f0d0b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a782e5-89zws\" (UID: \"a0c236a4-aee1-407f-85a3-49f13a1f0d0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a782e5-89zws" Oct 30 00:38:06.618746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.618671 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c236a4-aee1-407f-85a3-49f13a1f0d0b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a782e5-89zws\" (UID: \"a0c236a4-aee1-407f-85a3-49f13a1f0d0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a782e5-89zws" Oct 30 00:38:06.619528 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.619501 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c236a4-aee1-407f-85a3-49f13a1f0d0b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a782e5-89zws\" (UID: \"a0c236a4-aee1-407f-85a3-49f13a1f0d0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a782e5-89zws" Oct 30 00:38:06.621274 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.621249 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c236a4-aee1-407f-85a3-49f13a1f0d0b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a782e5-89zws\" (UID: \"a0c236a4-aee1-407f-85a3-49f13a1f0d0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a782e5-89zws" Oct 30 00:38:06.639174 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.639149 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8q2g\" (UniqueName: \"kubernetes.io/projected/a0c236a4-aee1-407f-85a3-49f13a1f0d0b-kube-api-access-g8q2g\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a782e5-89zws\" (UID: \"a0c236a4-aee1-407f-85a3-49f13a1f0d0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a782e5-89zws" Oct 30 00:38:06.675231 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.675204 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a782e5-89zws" Oct 30 00:38:06.921615 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.921529 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417" path="/var/lib/kubelet/pods/22ba3bbe-857d-4ec4-8ca7-2e7cfcbc1417/volumes" Oct 30 00:38:06.922189 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.922169 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdd8c42e-35c2-4141-afab-e400b23a9163" path="/var/lib/kubelet/pods/cdd8c42e-35c2-4141-afab-e400b23a9163/volumes" Oct 30 00:38:06.988871 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:06.988555 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a782e5-89zws"] Oct 30 00:38:07.476277 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:07.476237 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a782e5-89zws" event={"ID":"a0c236a4-aee1-407f-85a3-49f13a1f0d0b","Type":"ContainerStarted","Data":"a5d80a0725d5e123f8f1ef39275a9fa34f8f4eb4ba14fc4d8e83be7506d5d5dc"} Oct 30 00:38:07.476660 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:07.476301 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a782e5-89zws" event={"ID":"a0c236a4-aee1-407f-85a3-49f13a1f0d0b","Type":"ContainerStarted","Data":"e13101774d097ff9c8d9d07d747e5c892d152ed909e34f98333a0d663ddf1c9c"} Oct 30 00:38:07.513726 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:07.513677 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a782e5-89zws" podStartSLOduration=1.513662374 podStartE2EDuration="1.513662374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:07.510438561 +0000 UTC m=+2002.715939753" watchObservedRunningTime="2026-10-30 00:38:07.513662374 +0000 UTC m=+2002.719163588" Oct 30 00:38:07.814471 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:07.814374 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-29f3ca-6765676c44-vvgqp" podUID="d4c8106e-3826-4f4d-b267-93ecea650c90" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:38:08.173323 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:08.173238 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ecb34a-7ffdbf96c-8lgp2" podUID="aca30d4b-8350-4bfd-ba90-54e70dacf765" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:38:08.488977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:08.488855 5001 generic.go:356] "Generic (PLEG): container finished" podID="a0c236a4-aee1-407f-85a3-49f13a1f0d0b" containerID="a5d80a0725d5e123f8f1ef39275a9fa34f8f4eb4ba14fc4d8e83be7506d5d5dc" exitCode=0 Oct 30 00:38:08.488977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:08.488900 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a782e5-89zws" event={"ID":"a0c236a4-aee1-407f-85a3-49f13a1f0d0b","Type":"ContainerDied","Data":"a5d80a0725d5e123f8f1ef39275a9fa34f8f4eb4ba14fc4d8e83be7506d5d5dc"} Oct 30 00:38:09.196342 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:09.196306 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ecb34a-7c4f79cf86-6l8nz" podUID="0261228f-3733-4993-893e-43c742c7bd5e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:38:09.717747 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:09.717683 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a782e5-89zws" Oct 30 00:38:09.775692 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:09.775617 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0c236a4-aee1-407f-85a3-49f13a1f0d0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c236a4-aee1-407f-85a3-49f13a1f0d0b-maas-api-xks-ca\") pod \"a0c236a4-aee1-407f-85a3-49f13a1f0d0b\" (UID: \"a0c236a4-aee1-407f-85a3-49f13a1f0d0b\") " Oct 30 00:38:09.775877 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:09.775743 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0c236a4-aee1-407f-85a3-49f13a1f0d0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c236a4-aee1-407f-85a3-49f13a1f0d0b-maas-api-openshift-ca\") pod \"a0c236a4-aee1-407f-85a3-49f13a1f0d0b\" (UID: \"a0c236a4-aee1-407f-85a3-49f13a1f0d0b\") " Oct 30 00:38:09.775877 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:09.775782 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0c236a4-aee1-407f-85a3-49f13a1f0d0b-kube-api-access-g8q2g\" (UniqueName: \"kubernetes.io/projected/a0c236a4-aee1-407f-85a3-49f13a1f0d0b-kube-api-access-g8q2g\") pod \"a0c236a4-aee1-407f-85a3-49f13a1f0d0b\" (UID: \"a0c236a4-aee1-407f-85a3-49f13a1f0d0b\") " Oct 30 00:38:09.776485 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:09.776457 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0c236a4-aee1-407f-85a3-49f13a1f0d0b-maas-api-openshift-ca" pod "a0c236a4-aee1-407f-85a3-49f13a1f0d0b" (UID: "a0c236a4-aee1-407f-85a3-49f13a1f0d0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:38:09.778338 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:09.778305 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0c236a4-aee1-407f-85a3-49f13a1f0d0b-maas-api-xks-ca" pod "a0c236a4-aee1-407f-85a3-49f13a1f0d0b" (UID: "a0c236a4-aee1-407f-85a3-49f13a1f0d0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:38:09.778697 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:09.778662 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c236a4-aee1-407f-85a3-49f13a1f0d0b-kube-api-access-g8q2g" pod "a0c236a4-aee1-407f-85a3-49f13a1f0d0b" (UID: "a0c236a4-aee1-407f-85a3-49f13a1f0d0b"). InnerVolumeSpecName "kube-api-access-g8q2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:09.877735 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:09.877699 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c236a4-aee1-407f-85a3-49f13a1f0d0b-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:09.877735 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:09.877736 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c236a4-aee1-407f-85a3-49f13a1f0d0b-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:09.877952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:09.877750 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8q2g\" (UniqueName: \"kubernetes.io/projected/a0c236a4-aee1-407f-85a3-49f13a1f0d0b-kube-api-access-g8q2g\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:10.029294 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:10.029206 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e45580f4-lwvdl"] Oct 30 00:38:10.033801 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:10.033769 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e45580f4-lwvdl"] Oct 30 00:38:10.507448 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:10.507373 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a782e5-89zws" Oct 30 00:38:10.507614 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:10.507377 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a782e5-89zws" event={"ID":"a0c236a4-aee1-407f-85a3-49f13a1f0d0b","Type":"ContainerDied","Data":"e13101774d097ff9c8d9d07d747e5c892d152ed909e34f98333a0d663ddf1c9c"} Oct 30 00:38:10.507614 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:10.507498 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e13101774d097ff9c8d9d07d747e5c892d152ed909e34f98333a0d663ddf1c9c" Oct 30 00:38:10.974811 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:10.974768 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df407c8d-1211-43cc-be72-7096f158be35" path="/var/lib/kubelet/pods/df407c8d-1211-43cc-be72-7096f158be35/volumes" Oct 30 00:38:11.036114 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:11.036073 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-02665b-hzfvt"] Oct 30 00:38:11.036698 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:11.036681 5001 cpu_manager.go:414] "containerMap: removing container" podUID="a0c236a4-aee1-407f-85a3-49f13a1f0d0b" containerName="revoke-keys" Oct 30 00:38:11.036698 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:11.036696 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0c236a4-aee1-407f-85a3-49f13a1f0d0b" containerName="revoke-keys" Oct 30 00:38:11.036860 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:11.036832 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0c236a4-aee1-407f-85a3-49f13a1f0d0b" containerName="revoke-keys" Oct 30 00:38:11.041679 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:11.041656 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-02665b-hzfvt" Oct 30 00:38:11.056575 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:11.056503 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:38:11.056575 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:11.056509 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:38:11.074508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:11.074481 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-02665b-hzfvt"] Oct 30 00:38:11.082087 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:11.082057 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:38:11.083142 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:11.083122 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:38:11.098832 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:11.098807 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d32d4d5-a424-48e4-94aa-fcf81482ebef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-02665b-hzfvt\" (UID: \"8d32d4d5-a424-48e4-94aa-fcf81482ebef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-02665b-hzfvt" Oct 30 00:38:11.098961 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:11.098938 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76gpw\" (UniqueName: \"kubernetes.io/projected/8d32d4d5-a424-48e4-94aa-fcf81482ebef-kube-api-access-76gpw\") pod \"maas-api-revoke-keys-e2e-worker-w4-02665b-hzfvt\" (UID: \"8d32d4d5-a424-48e4-94aa-fcf81482ebef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-02665b-hzfvt" Oct 30 00:38:11.099055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:11.099036 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d32d4d5-a424-48e4-94aa-fcf81482ebef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-02665b-hzfvt\" (UID: \"8d32d4d5-a424-48e4-94aa-fcf81482ebef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-02665b-hzfvt" Oct 30 00:38:11.200323 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:11.200291 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d32d4d5-a424-48e4-94aa-fcf81482ebef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-02665b-hzfvt\" (UID: \"8d32d4d5-a424-48e4-94aa-fcf81482ebef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-02665b-hzfvt" Oct 30 00:38:11.200500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:11.200357 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76gpw\" (UniqueName: \"kubernetes.io/projected/8d32d4d5-a424-48e4-94aa-fcf81482ebef-kube-api-access-76gpw\") pod \"maas-api-revoke-keys-e2e-worker-w4-02665b-hzfvt\" (UID: \"8d32d4d5-a424-48e4-94aa-fcf81482ebef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-02665b-hzfvt" Oct 30 00:38:11.200500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:11.200415 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d32d4d5-a424-48e4-94aa-fcf81482ebef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-02665b-hzfvt\" (UID: \"8d32d4d5-a424-48e4-94aa-fcf81482ebef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-02665b-hzfvt" Oct 30 00:38:11.200964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:11.200939 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d32d4d5-a424-48e4-94aa-fcf81482ebef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-02665b-hzfvt\" (UID: \"8d32d4d5-a424-48e4-94aa-fcf81482ebef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-02665b-hzfvt" Oct 30 00:38:11.202829 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:11.202808 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d32d4d5-a424-48e4-94aa-fcf81482ebef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-02665b-hzfvt\" (UID: \"8d32d4d5-a424-48e4-94aa-fcf81482ebef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-02665b-hzfvt" Oct 30 00:38:11.249880 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:11.249814 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76gpw\" (UniqueName: \"kubernetes.io/projected/8d32d4d5-a424-48e4-94aa-fcf81482ebef-kube-api-access-76gpw\") pod \"maas-api-revoke-keys-e2e-worker-w4-02665b-hzfvt\" (UID: \"8d32d4d5-a424-48e4-94aa-fcf81482ebef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-02665b-hzfvt" Oct 30 00:38:11.357822 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:11.357790 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-02665b-hzfvt" Oct 30 00:38:11.675933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:11.674570 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-02665b-hzfvt"] Oct 30 00:38:12.526003 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:12.525963 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-02665b-hzfvt" event={"ID":"8d32d4d5-a424-48e4-94aa-fcf81482ebef","Type":"ContainerStarted","Data":"b5ad0a5fa106e1ab82886edba9f4b21e6f4eb7a3ea076f648c8052852cf1d361"} Oct 30 00:38:12.526395 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:12.526019 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-02665b-hzfvt" event={"ID":"8d32d4d5-a424-48e4-94aa-fcf81482ebef","Type":"ContainerStarted","Data":"bf7556250bced0b72a9bd4f973a10d9f20820995489b39fb144ed0666af0837f"} Oct 30 00:38:12.562497 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:12.562444 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-02665b-hzfvt" podStartSLOduration=1.5624301090000001 podStartE2EDuration="1.562430109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:12.56097551 +0000 UTC m=+2007.766476694" watchObservedRunningTime="2026-10-30 00:38:12.562430109 +0000 UTC m=+2007.767931349" Oct 30 00:38:13.535785 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:13.535750 5001 generic.go:356] "Generic (PLEG): container finished" podID="8d32d4d5-a424-48e4-94aa-fcf81482ebef" containerID="b5ad0a5fa106e1ab82886edba9f4b21e6f4eb7a3ea076f648c8052852cf1d361" exitCode=0 Oct 30 00:38:13.536217 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:13.535837 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-02665b-hzfvt" event={"ID":"8d32d4d5-a424-48e4-94aa-fcf81482ebef","Type":"ContainerDied","Data":"b5ad0a5fa106e1ab82886edba9f4b21e6f4eb7a3ea076f648c8052852cf1d361"} Oct 30 00:38:14.030940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:14.030874 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c061895-7m288"] Oct 30 00:38:14.040905 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:14.040876 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c061895-7m288"] Oct 30 00:38:14.714723 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:14.714697 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-02665b-hzfvt" Oct 30 00:38:14.744614 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:14.744585 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d32d4d5-a424-48e4-94aa-fcf81482ebef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d32d4d5-a424-48e4-94aa-fcf81482ebef-maas-api-openshift-ca\") pod \"8d32d4d5-a424-48e4-94aa-fcf81482ebef\" (UID: \"8d32d4d5-a424-48e4-94aa-fcf81482ebef\") " Oct 30 00:38:14.744814 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:14.744801 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d32d4d5-a424-48e4-94aa-fcf81482ebef-kube-api-access-76gpw\" (UniqueName: \"kubernetes.io/projected/8d32d4d5-a424-48e4-94aa-fcf81482ebef-kube-api-access-76gpw\") pod \"8d32d4d5-a424-48e4-94aa-fcf81482ebef\" (UID: \"8d32d4d5-a424-48e4-94aa-fcf81482ebef\") " Oct 30 00:38:14.744872 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:14.744860 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d32d4d5-a424-48e4-94aa-fcf81482ebef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d32d4d5-a424-48e4-94aa-fcf81482ebef-maas-api-xks-ca\") pod \"8d32d4d5-a424-48e4-94aa-fcf81482ebef\" (UID: \"8d32d4d5-a424-48e4-94aa-fcf81482ebef\") " Oct 30 00:38:14.744967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:14.744941 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d32d4d5-a424-48e4-94aa-fcf81482ebef-maas-api-openshift-ca" pod "8d32d4d5-a424-48e4-94aa-fcf81482ebef" (UID: "8d32d4d5-a424-48e4-94aa-fcf81482ebef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:38:14.745287 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:14.745268 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d32d4d5-a424-48e4-94aa-fcf81482ebef-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:14.746938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:14.746897 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d32d4d5-a424-48e4-94aa-fcf81482ebef-kube-api-access-76gpw" pod "8d32d4d5-a424-48e4-94aa-fcf81482ebef" (UID: "8d32d4d5-a424-48e4-94aa-fcf81482ebef"). InnerVolumeSpecName "kube-api-access-76gpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:14.747042 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:14.746931 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d32d4d5-a424-48e4-94aa-fcf81482ebef-maas-api-xks-ca" pod "8d32d4d5-a424-48e4-94aa-fcf81482ebef" (UID: "8d32d4d5-a424-48e4-94aa-fcf81482ebef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:38:14.846977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:14.846873 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76gpw\" (UniqueName: \"kubernetes.io/projected/8d32d4d5-a424-48e4-94aa-fcf81482ebef-kube-api-access-76gpw\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:14.846977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:14.846901 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d32d4d5-a424-48e4-94aa-fcf81482ebef-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:14.919292 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:14.919257 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf8f0e8a-689f-42cd-9e12-106da73dd5b1" path="/var/lib/kubelet/pods/bf8f0e8a-689f-42cd-9e12-106da73dd5b1/volumes" Oct 30 00:38:15.553862 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:15.553827 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-02665b-hzfvt" Oct 30 00:38:15.554068 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:15.553859 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-02665b-hzfvt" event={"ID":"8d32d4d5-a424-48e4-94aa-fcf81482ebef","Type":"ContainerDied","Data":"bf7556250bced0b72a9bd4f973a10d9f20820995489b39fb144ed0666af0837f"} Oct 30 00:38:15.554068 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:15.553905 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf7556250bced0b72a9bd4f973a10d9f20820995489b39fb144ed0666af0837f" Oct 30 00:38:15.731692 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:15.731657 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s"] Oct 30 00:38:16.784939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:16.784879 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a782e5-56b6d8bf96-8pss5"] Oct 30 00:38:16.815186 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:16.815157 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a782e5-7dddfb8f55-tcmx6"] Oct 30 00:38:17.323485 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:38:17.323446 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:38:17.814628 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:17.814589 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-29f3ca-6765676c44-vvgqp" podUID="d4c8106e-3826-4f4d-b267-93ecea650c90" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:38:18.173632 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:18.173552 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ecb34a-7ffdbf96c-8lgp2" podUID="aca30d4b-8350-4bfd-ba90-54e70dacf765" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:38:18.891902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:18.891864 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-29f3ca-dvs48"] Oct 30 00:38:18.901246 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:18.901220 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-29f3ca-dvs48"] Oct 30 00:38:18.919445 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:18.919415 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a76339ab-86f8-4e69-8844-730227c58fd7" path="/var/lib/kubelet/pods/a76339ab-86f8-4e69-8844-730227c58fd7/volumes" Oct 30 00:38:19.195993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:19.195881 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ecb34a-7c4f79cf86-6l8nz" podUID="0261228f-3733-4993-893e-43c742c7bd5e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:38:21.079464 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.079418 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz" podUID="2326ccd2-e034-479e-9ffd-6366f1327831" containerName="main" containerID="cri-o://bff93ef65e145b762c31958f3fa5ecab1e14ec53f6dc81aada2019e5f7ca2a77" gracePeriod=45 Oct 30 00:38:21.248070 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.248032 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz" podUID="2326ccd2-e034-479e-9ffd-6366f1327831" containerName="main" probeResult="failure" output="Get \"https://10.132.1.31:8000/ready\": dial tcp 10.132.1.31:8000: connect: connection refused" Oct 30 00:38:21.345397 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.345335 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz" Oct 30 00:38:21.435661 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.435625 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2326ccd2-e034-479e-9ffd-6366f1327831-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2326ccd2-e034-479e-9ffd-6366f1327831-model-cache\") pod \"2326ccd2-e034-479e-9ffd-6366f1327831\" (UID: \"2326ccd2-e034-479e-9ffd-6366f1327831\") " Oct 30 00:38:21.435852 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.435686 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2326ccd2-e034-479e-9ffd-6366f1327831-tls-certs\" (UniqueName: \"kubernetes.io/secret/2326ccd2-e034-479e-9ffd-6366f1327831-tls-certs\") pod \"2326ccd2-e034-479e-9ffd-6366f1327831\" (UID: \"2326ccd2-e034-479e-9ffd-6366f1327831\") " Oct 30 00:38:21.435852 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.435733 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2326ccd2-e034-479e-9ffd-6366f1327831-kube-api-access-456zj\" (UniqueName: \"kubernetes.io/projected/2326ccd2-e034-479e-9ffd-6366f1327831-kube-api-access-456zj\") pod \"2326ccd2-e034-479e-9ffd-6366f1327831\" (UID: \"2326ccd2-e034-479e-9ffd-6366f1327831\") " Oct 30 00:38:21.435852 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.435761 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2326ccd2-e034-479e-9ffd-6366f1327831-home\" (UniqueName: \"kubernetes.io/empty-dir/2326ccd2-e034-479e-9ffd-6366f1327831-home\") pod \"2326ccd2-e034-479e-9ffd-6366f1327831\" (UID: \"2326ccd2-e034-479e-9ffd-6366f1327831\") " Oct 30 00:38:21.435852 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.435829 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2326ccd2-e034-479e-9ffd-6366f1327831-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2326ccd2-e034-479e-9ffd-6366f1327831-tmp-dir\") pod \"2326ccd2-e034-479e-9ffd-6366f1327831\" (UID: \"2326ccd2-e034-479e-9ffd-6366f1327831\") " Oct 30 00:38:21.436144 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.435946 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2326ccd2-e034-479e-9ffd-6366f1327831-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2326ccd2-e034-479e-9ffd-6366f1327831-dshm\") pod \"2326ccd2-e034-479e-9ffd-6366f1327831\" (UID: \"2326ccd2-e034-479e-9ffd-6366f1327831\") " Oct 30 00:38:21.436144 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.435998 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2326ccd2-e034-479e-9ffd-6366f1327831-model-cache" pod "2326ccd2-e034-479e-9ffd-6366f1327831" (UID: "2326ccd2-e034-479e-9ffd-6366f1327831"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:21.436247 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.436118 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2326ccd2-e034-479e-9ffd-6366f1327831-home" pod "2326ccd2-e034-479e-9ffd-6366f1327831" (UID: "2326ccd2-e034-479e-9ffd-6366f1327831"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:21.436247 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.436200 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2326ccd2-e034-479e-9ffd-6366f1327831-tmp-dir" pod "2326ccd2-e034-479e-9ffd-6366f1327831" (UID: "2326ccd2-e034-479e-9ffd-6366f1327831"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:21.436660 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.436635 5001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2326ccd2-e034-479e-9ffd-6366f1327831-home\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:21.436660 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.436656 5001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2326ccd2-e034-479e-9ffd-6366f1327831-tmp-dir\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:21.436777 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.436672 5001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2326ccd2-e034-479e-9ffd-6366f1327831-model-cache\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:21.438168 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.438139 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2326ccd2-e034-479e-9ffd-6366f1327831-dshm" pod "2326ccd2-e034-479e-9ffd-6366f1327831" (UID: "2326ccd2-e034-479e-9ffd-6366f1327831"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:21.438281 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.438201 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2326ccd2-e034-479e-9ffd-6366f1327831-tls-certs" pod "2326ccd2-e034-479e-9ffd-6366f1327831" (UID: "2326ccd2-e034-479e-9ffd-6366f1327831"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:38:21.438281 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.438225 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2326ccd2-e034-479e-9ffd-6366f1327831-kube-api-access-456zj" pod "2326ccd2-e034-479e-9ffd-6366f1327831" (UID: "2326ccd2-e034-479e-9ffd-6366f1327831"). InnerVolumeSpecName "kube-api-access-456zj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:21.476743 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.476715 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-02665b-c768b5f9b-dpvxb"] Oct 30 00:38:21.538033 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.538000 5001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2326ccd2-e034-479e-9ffd-6366f1327831-tls-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:21.538033 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.538030 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-456zj\" (UniqueName: \"kubernetes.io/projected/2326ccd2-e034-479e-9ffd-6366f1327831-kube-api-access-456zj\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:21.538033 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.538040 5001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2326ccd2-e034-479e-9ffd-6366f1327831-dshm\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:21.624864 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.624768 5001 generic.go:356] "Generic (PLEG): container finished" podID="2326ccd2-e034-479e-9ffd-6366f1327831" containerID="bff93ef65e145b762c31958f3fa5ecab1e14ec53f6dc81aada2019e5f7ca2a77" exitCode=0 Oct 30 00:38:21.625165 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.624902 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz" Oct 30 00:38:21.625165 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.624969 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz" event={"ID":"2326ccd2-e034-479e-9ffd-6366f1327831","Type":"ContainerDied","Data":"bff93ef65e145b762c31958f3fa5ecab1e14ec53f6dc81aada2019e5f7ca2a77"} Oct 30 00:38:21.625165 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.625004 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz" event={"ID":"2326ccd2-e034-479e-9ffd-6366f1327831","Type":"ContainerDied","Data":"fc15c4cbc0a31a5a5842de879fdb6807095cf988d68c906314e4742f9952a620"} Oct 30 00:38:21.625165 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.625027 5001 scope.go:122] "RemoveContainer" containerID="bff93ef65e145b762c31958f3fa5ecab1e14ec53f6dc81aada2019e5f7ca2a77" Oct 30 00:38:21.640442 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.640419 5001 scope.go:122] "RemoveContainer" containerID="bff93ef65e145b762c31958f3fa5ecab1e14ec53f6dc81aada2019e5f7ca2a77" Oct 30 00:38:21.640753 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:38:21.640727 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bff93ef65e145b762c31958f3fa5ecab1e14ec53f6dc81aada2019e5f7ca2a77\": container with ID starting with bff93ef65e145b762c31958f3fa5ecab1e14ec53f6dc81aada2019e5f7ca2a77 not found: ID does not exist" containerID="bff93ef65e145b762c31958f3fa5ecab1e14ec53f6dc81aada2019e5f7ca2a77" Oct 30 00:38:21.640856 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.640764 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bff93ef65e145b762c31958f3fa5ecab1e14ec53f6dc81aada2019e5f7ca2a77"} err="failed to get container status \"bff93ef65e145b762c31958f3fa5ecab1e14ec53f6dc81aada2019e5f7ca2a77\": rpc error: code = NotFound desc = could not find container \"bff93ef65e145b762c31958f3fa5ecab1e14ec53f6dc81aada2019e5f7ca2a77\": container with ID starting with bff93ef65e145b762c31958f3fa5ecab1e14ec53f6dc81aada2019e5f7ca2a77 not found: ID does not exist" Oct 30 00:38:21.672491 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.672465 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz"] Oct 30 00:38:21.678039 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.678017 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-05c6b2/auth-test-model-05c6b2-kserve-db87bf5b5-2tmzz"] Oct 30 00:38:21.785887 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.785814 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a782e5-56b6d8bf96-8pss5" podUID="d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea" containerName="payload-processing" containerID="cri-o://a687cb927a21b9784bf2a4f2d63fbd9a1ca9aea48c31f4791fca9ac3041fa850" gracePeriod=25 Oct 30 00:38:21.815639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.815598 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a782e5-7dddfb8f55-tcmx6" podUID="3a97385f-5ca1-41db-9a2a-66cbc70baad2" containerName="payload-pre-processing" containerID="cri-o://4edf7887e39c5427a700c76a4d8230032c53349f5532230dd1762d3e302bd451" gracePeriod=25 Oct 30 00:38:21.867099 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:21.866993 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx"] Oct 30 00:38:22.920123 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:22.920097 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2326ccd2-e034-479e-9ffd-6366f1327831" path="/var/lib/kubelet/pods/2326ccd2-e034-479e-9ffd-6366f1327831/volumes" Oct 30 00:38:22.968553 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:22.968494 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a782e5-89zws"] Oct 30 00:38:22.973696 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:22.973667 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a782e5-89zws"] Oct 30 00:38:23.101097 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:23.101065 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ecb34a-7c4f79cf86-6l8nz" Oct 30 00:38:23.137438 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:23.137409 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ecb34a-7ffdbf96c-8lgp2" Oct 30 00:38:23.167147 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:23.167110 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0261228f-3733-4993-893e-43c742c7bd5e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0261228f-3733-4993-893e-43c742c7bd5e-plugins-config-volume\") pod \"0261228f-3733-4993-893e-43c742c7bd5e\" (UID: \"0261228f-3733-4993-893e-43c742c7bd5e\") " Oct 30 00:38:23.167281 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:23.167192 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0261228f-3733-4993-893e-43c742c7bd5e-kube-api-access-v7d7g\" (UniqueName: \"kubernetes.io/projected/0261228f-3733-4993-893e-43c742c7bd5e-kube-api-access-v7d7g\") pod \"0261228f-3733-4993-893e-43c742c7bd5e\" (UID: \"0261228f-3733-4993-893e-43c742c7bd5e\") " Oct 30 00:38:23.167574 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:23.167547 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0261228f-3733-4993-893e-43c742c7bd5e-plugins-config-volume" pod "0261228f-3733-4993-893e-43c742c7bd5e" (UID: "0261228f-3733-4993-893e-43c742c7bd5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:38:23.167921 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:23.167887 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0261228f-3733-4993-893e-43c742c7bd5e-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:23.169924 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:23.169876 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0261228f-3733-4993-893e-43c742c7bd5e-kube-api-access-v7d7g" pod "0261228f-3733-4993-893e-43c742c7bd5e" (UID: "0261228f-3733-4993-893e-43c742c7bd5e"). InnerVolumeSpecName "kube-api-access-v7d7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:23.268632 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:23.268595 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aca30d4b-8350-4bfd-ba90-54e70dacf765-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca30d4b-8350-4bfd-ba90-54e70dacf765-plugins-config-volume\") pod \"aca30d4b-8350-4bfd-ba90-54e70dacf765\" (UID: \"aca30d4b-8350-4bfd-ba90-54e70dacf765\") " Oct 30 00:38:23.268775 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:23.268728 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aca30d4b-8350-4bfd-ba90-54e70dacf765-kube-api-access-lbvr7\" (UniqueName: \"kubernetes.io/projected/aca30d4b-8350-4bfd-ba90-54e70dacf765-kube-api-access-lbvr7\") pod \"aca30d4b-8350-4bfd-ba90-54e70dacf765\" (UID: \"aca30d4b-8350-4bfd-ba90-54e70dacf765\") " Oct 30 00:38:23.269086 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:23.269065 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aca30d4b-8350-4bfd-ba90-54e70dacf765-plugins-config-volume" pod "aca30d4b-8350-4bfd-ba90-54e70dacf765" (UID: "aca30d4b-8350-4bfd-ba90-54e70dacf765"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:38:23.269265 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:23.269250 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7d7g\" (UniqueName: \"kubernetes.io/projected/0261228f-3733-4993-893e-43c742c7bd5e-kube-api-access-v7d7g\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:23.269307 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:23.269269 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca30d4b-8350-4bfd-ba90-54e70dacf765-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:23.270797 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:23.270774 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aca30d4b-8350-4bfd-ba90-54e70dacf765-kube-api-access-lbvr7" pod "aca30d4b-8350-4bfd-ba90-54e70dacf765" (UID: "aca30d4b-8350-4bfd-ba90-54e70dacf765"). InnerVolumeSpecName "kube-api-access-lbvr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:23.370171 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:23.370102 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbvr7\" (UniqueName: \"kubernetes.io/projected/aca30d4b-8350-4bfd-ba90-54e70dacf765-kube-api-access-lbvr7\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:23.695971 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:23.695860 5001 generic.go:356] "Generic (PLEG): container finished" podID="0261228f-3733-4993-893e-43c742c7bd5e" containerID="f333c84868e04ee11a4d80dc0251fa3008a4c0ca379f0ce0a97a7166ba0a9e2a" exitCode=0 Oct 30 00:38:23.695971 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:23.695934 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ecb34a-7c4f79cf86-6l8nz" Oct 30 00:38:23.695971 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:23.695949 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ecb34a-7c4f79cf86-6l8nz" event={"ID":"0261228f-3733-4993-893e-43c742c7bd5e","Type":"ContainerDied","Data":"f333c84868e04ee11a4d80dc0251fa3008a4c0ca379f0ce0a97a7166ba0a9e2a"} Oct 30 00:38:23.696228 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:23.696008 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ecb34a-7c4f79cf86-6l8nz" event={"ID":"0261228f-3733-4993-893e-43c742c7bd5e","Type":"ContainerDied","Data":"25e48c2a3fc075d6c0f95cbed64bb6e6636e5d00bc2635b77952d4f7b5ee6711"} Oct 30 00:38:23.696228 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:23.696032 5001 scope.go:122] "RemoveContainer" containerID="f333c84868e04ee11a4d80dc0251fa3008a4c0ca379f0ce0a97a7166ba0a9e2a" Oct 30 00:38:23.697699 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:23.697673 5001 generic.go:356] "Generic (PLEG): container finished" podID="aca30d4b-8350-4bfd-ba90-54e70dacf765" containerID="2d9c1cf0fefda535b52b87921c9e3752803c6e1de1a90bc46be95d007cd32f38" exitCode=0 Oct 30 00:38:23.697839 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:23.697701 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ecb34a-7ffdbf96c-8lgp2" event={"ID":"aca30d4b-8350-4bfd-ba90-54e70dacf765","Type":"ContainerDied","Data":"2d9c1cf0fefda535b52b87921c9e3752803c6e1de1a90bc46be95d007cd32f38"} Oct 30 00:38:23.697839 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:23.697723 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ecb34a-7ffdbf96c-8lgp2" Oct 30 00:38:23.697839 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:23.697748 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ecb34a-7ffdbf96c-8lgp2" event={"ID":"aca30d4b-8350-4bfd-ba90-54e70dacf765","Type":"ContainerDied","Data":"f5c1bc9ee34f9a08811dbd6279cde733d1e43ec914fea07bf9e7f9fa962bd8ab"} Oct 30 00:38:23.710156 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:23.710132 5001 scope.go:122] "RemoveContainer" containerID="f333c84868e04ee11a4d80dc0251fa3008a4c0ca379f0ce0a97a7166ba0a9e2a" Oct 30 00:38:23.710445 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:38:23.710425 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f333c84868e04ee11a4d80dc0251fa3008a4c0ca379f0ce0a97a7166ba0a9e2a\": container with ID starting with f333c84868e04ee11a4d80dc0251fa3008a4c0ca379f0ce0a97a7166ba0a9e2a not found: ID does not exist" containerID="f333c84868e04ee11a4d80dc0251fa3008a4c0ca379f0ce0a97a7166ba0a9e2a" Oct 30 00:38:23.710513 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:23.710452 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f333c84868e04ee11a4d80dc0251fa3008a4c0ca379f0ce0a97a7166ba0a9e2a"} err="failed to get container status \"f333c84868e04ee11a4d80dc0251fa3008a4c0ca379f0ce0a97a7166ba0a9e2a\": rpc error: code = NotFound desc = could not find container \"f333c84868e04ee11a4d80dc0251fa3008a4c0ca379f0ce0a97a7166ba0a9e2a\": container with ID starting with f333c84868e04ee11a4d80dc0251fa3008a4c0ca379f0ce0a97a7166ba0a9e2a not found: ID does not exist" Oct 30 00:38:23.710513 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:23.710475 5001 scope.go:122] "RemoveContainer" containerID="2d9c1cf0fefda535b52b87921c9e3752803c6e1de1a90bc46be95d007cd32f38" Oct 30 00:38:23.721711 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:23.721687 5001 scope.go:122] "RemoveContainer" containerID="2d9c1cf0fefda535b52b87921c9e3752803c6e1de1a90bc46be95d007cd32f38" Oct 30 00:38:23.722018 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:38:23.721998 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d9c1cf0fefda535b52b87921c9e3752803c6e1de1a90bc46be95d007cd32f38\": container with ID starting with 2d9c1cf0fefda535b52b87921c9e3752803c6e1de1a90bc46be95d007cd32f38 not found: ID does not exist" containerID="2d9c1cf0fefda535b52b87921c9e3752803c6e1de1a90bc46be95d007cd32f38" Oct 30 00:38:23.722088 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:23.722029 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d9c1cf0fefda535b52b87921c9e3752803c6e1de1a90bc46be95d007cd32f38"} err="failed to get container status \"2d9c1cf0fefda535b52b87921c9e3752803c6e1de1a90bc46be95d007cd32f38\": rpc error: code = NotFound desc = could not find container \"2d9c1cf0fefda535b52b87921c9e3752803c6e1de1a90bc46be95d007cd32f38\": container with ID starting with 2d9c1cf0fefda535b52b87921c9e3752803c6e1de1a90bc46be95d007cd32f38 not found: ID does not exist" Oct 30 00:38:23.744545 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:23.744518 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-ecb34a-7c4f79cf86-6l8nz"] Oct 30 00:38:23.752956 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:23.752932 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-ecb34a-7c4f79cf86-6l8nz"] Oct 30 00:38:23.787887 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:23.787852 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ecb34a-7ffdbf96c-8lgp2"] Oct 30 00:38:23.792317 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:23.792290 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ecb34a-7ffdbf96c-8lgp2"] Oct 30 00:38:24.358486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.358451 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a307fa79-fmcgz"] Oct 30 00:38:24.359141 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.359122 5001 cpu_manager.go:414] "containerMap: removing container" podUID="0261228f-3733-4993-893e-43c742c7bd5e" containerName="payload-processing" Oct 30 00:38:24.359141 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.359142 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0261228f-3733-4993-893e-43c742c7bd5e" containerName="payload-processing" Oct 30 00:38:24.359252 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.359155 5001 cpu_manager.go:414] "containerMap: removing container" podUID="8d32d4d5-a424-48e4-94aa-fcf81482ebef" containerName="revoke-keys" Oct 30 00:38:24.359252 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.359161 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d32d4d5-a424-48e4-94aa-fcf81482ebef" containerName="revoke-keys" Oct 30 00:38:24.359252 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.359170 5001 cpu_manager.go:414] "containerMap: removing container" podUID="aca30d4b-8350-4bfd-ba90-54e70dacf765" containerName="payload-pre-processing" Oct 30 00:38:24.359252 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.359175 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aca30d4b-8350-4bfd-ba90-54e70dacf765" containerName="payload-pre-processing" Oct 30 00:38:24.359252 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.359189 5001 cpu_manager.go:414] "containerMap: removing container" podUID="2326ccd2-e034-479e-9ffd-6366f1327831" containerName="main" Oct 30 00:38:24.359252 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.359194 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2326ccd2-e034-479e-9ffd-6366f1327831" containerName="main" Oct 30 00:38:24.359439 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.359292 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0261228f-3733-4993-893e-43c742c7bd5e" containerName="payload-processing" Oct 30 00:38:24.359439 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.359300 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d32d4d5-a424-48e4-94aa-fcf81482ebef" containerName="revoke-keys" Oct 30 00:38:24.359439 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.359307 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="aca30d4b-8350-4bfd-ba90-54e70dacf765" containerName="payload-pre-processing" Oct 30 00:38:24.359439 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.359313 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2326ccd2-e034-479e-9ffd-6366f1327831" containerName="main" Oct 30 00:38:24.364528 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.364506 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a307fa79-fmcgz" Oct 30 00:38:24.373058 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.373037 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc72f0e2-k6fcc"] Oct 30 00:38:24.377226 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.377206 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc72f0e2-k6fcc" Oct 30 00:38:24.378206 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.378187 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:38:24.396149 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.387828 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:38:24.396149 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.387850 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:38:24.396149 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.387931 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:38:24.396149 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.395456 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a307fa79-fmcgz"] Oct 30 00:38:24.412855 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.412802 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc72f0e2-k6fcc"] Oct 30 00:38:24.483382 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.483348 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc72f0e2-k6fcc\" (UID: \"2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc72f0e2-k6fcc" Oct 30 00:38:24.483532 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.483484 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc72f0e2-k6fcc\" (UID: \"2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc72f0e2-k6fcc" Oct 30 00:38:24.483532 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.483516 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4b18928-3805-413b-be9c-7732bc0072a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a307fa79-fmcgz\" (UID: \"b4b18928-3805-413b-be9c-7732bc0072a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a307fa79-fmcgz" Oct 30 00:38:24.483602 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.483578 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqkw4\" (UniqueName: \"kubernetes.io/projected/2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1-kube-api-access-fqkw4\") pod \"maas-api-revoke-sub-cc72f0e2-k6fcc\" (UID: \"2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc72f0e2-k6fcc" Oct 30 00:38:24.483694 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.483674 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4b18928-3805-413b-be9c-7732bc0072a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a307fa79-fmcgz\" (UID: \"b4b18928-3805-413b-be9c-7732bc0072a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a307fa79-fmcgz" Oct 30 00:38:24.483792 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.483778 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2jh5\" (UniqueName: \"kubernetes.io/projected/b4b18928-3805-413b-be9c-7732bc0072a9-kube-api-access-f2jh5\") pod \"maas-api-revoke-sub-a307fa79-fmcgz\" (UID: \"b4b18928-3805-413b-be9c-7732bc0072a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a307fa79-fmcgz" Oct 30 00:38:24.585101 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.585070 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqkw4\" (UniqueName: \"kubernetes.io/projected/2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1-kube-api-access-fqkw4\") pod \"maas-api-revoke-sub-cc72f0e2-k6fcc\" (UID: \"2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc72f0e2-k6fcc" Oct 30 00:38:24.585275 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.585116 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4b18928-3805-413b-be9c-7732bc0072a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a307fa79-fmcgz\" (UID: \"b4b18928-3805-413b-be9c-7732bc0072a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a307fa79-fmcgz" Oct 30 00:38:24.585275 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.585182 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2jh5\" (UniqueName: \"kubernetes.io/projected/b4b18928-3805-413b-be9c-7732bc0072a9-kube-api-access-f2jh5\") pod \"maas-api-revoke-sub-a307fa79-fmcgz\" (UID: \"b4b18928-3805-413b-be9c-7732bc0072a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a307fa79-fmcgz" Oct 30 00:38:24.585370 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.585307 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc72f0e2-k6fcc\" (UID: \"2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc72f0e2-k6fcc" Oct 30 00:38:24.585479 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.585462 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc72f0e2-k6fcc\" (UID: \"2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc72f0e2-k6fcc" Oct 30 00:38:24.585517 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.585493 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4b18928-3805-413b-be9c-7732bc0072a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a307fa79-fmcgz\" (UID: \"b4b18928-3805-413b-be9c-7732bc0072a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a307fa79-fmcgz" Oct 30 00:38:24.586121 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.586096 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc72f0e2-k6fcc\" (UID: \"2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc72f0e2-k6fcc" Oct 30 00:38:24.586121 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.586114 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4b18928-3805-413b-be9c-7732bc0072a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a307fa79-fmcgz\" (UID: \"b4b18928-3805-413b-be9c-7732bc0072a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a307fa79-fmcgz" Oct 30 00:38:24.587620 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.587602 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4b18928-3805-413b-be9c-7732bc0072a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a307fa79-fmcgz\" (UID: \"b4b18928-3805-413b-be9c-7732bc0072a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a307fa79-fmcgz" Oct 30 00:38:24.587682 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.587629 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc72f0e2-k6fcc\" (UID: \"2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc72f0e2-k6fcc" Oct 30 00:38:24.648891 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.648816 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2jh5\" (UniqueName: \"kubernetes.io/projected/b4b18928-3805-413b-be9c-7732bc0072a9-kube-api-access-f2jh5\") pod \"maas-api-revoke-sub-a307fa79-fmcgz\" (UID: \"b4b18928-3805-413b-be9c-7732bc0072a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a307fa79-fmcgz" Oct 30 00:38:24.649104 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.649080 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqkw4\" (UniqueName: \"kubernetes.io/projected/2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1-kube-api-access-fqkw4\") pod \"maas-api-revoke-sub-cc72f0e2-k6fcc\" (UID: \"2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc72f0e2-k6fcc" Oct 30 00:38:24.679453 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.679429 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a307fa79-fmcgz" Oct 30 00:38:24.693842 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.693819 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc72f0e2-k6fcc" Oct 30 00:38:24.919827 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.919747 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0261228f-3733-4993-893e-43c742c7bd5e" path="/var/lib/kubelet/pods/0261228f-3733-4993-893e-43c742c7bd5e/volumes" Oct 30 00:38:24.920344 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.920322 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0c236a4-aee1-407f-85a3-49f13a1f0d0b" path="/var/lib/kubelet/pods/a0c236a4-aee1-407f-85a3-49f13a1f0d0b/volumes" Oct 30 00:38:24.920760 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.920745 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aca30d4b-8350-4bfd-ba90-54e70dacf765" path="/var/lib/kubelet/pods/aca30d4b-8350-4bfd-ba90-54e70dacf765/volumes" Oct 30 00:38:24.991431 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:38:24.991395 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4b18928_3805_413b_be9c_7732bc0072a9.slice/crio-4e2f3ade0ab2b9f290ebc623593d714df6364e5a040471bff01bc5b6f3f5346a WatchSource:0}: Error finding container 4e2f3ade0ab2b9f290ebc623593d714df6364e5a040471bff01bc5b6f3f5346a: Status 404 returned error can't find the container with id 4e2f3ade0ab2b9f290ebc623593d714df6364e5a040471bff01bc5b6f3f5346a Oct 30 00:38:24.992476 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:24.992453 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a307fa79-fmcgz"] Oct 30 00:38:25.015613 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:38:25.015587 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b1eb6e6_e4ce_44dc_b2ec_25a887349bd1.slice/crio-2c088292e1003ca783d64a82399e13aadfbe845f6bfa23be4758dd28538e0352 WatchSource:0}: Error finding container 2c088292e1003ca783d64a82399e13aadfbe845f6bfa23be4758dd28538e0352: Status 404 returned error can't find the container with id 2c088292e1003ca783d64a82399e13aadfbe845f6bfa23be4758dd28538e0352 Oct 30 00:38:25.018521 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:25.018493 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc72f0e2-k6fcc"] Oct 30 00:38:25.252998 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:25.252973 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-29f3ca-6765676c44-vvgqp" Oct 30 00:38:25.395472 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:25.395437 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4c8106e-3826-4f4d-b267-93ecea650c90-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4c8106e-3826-4f4d-b267-93ecea650c90-plugins-config-volume\") pod \"d4c8106e-3826-4f4d-b267-93ecea650c90\" (UID: \"d4c8106e-3826-4f4d-b267-93ecea650c90\") " Oct 30 00:38:25.395897 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:25.395632 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4c8106e-3826-4f4d-b267-93ecea650c90-kube-api-access-ql6cc\" (UniqueName: \"kubernetes.io/projected/d4c8106e-3826-4f4d-b267-93ecea650c90-kube-api-access-ql6cc\") pod \"d4c8106e-3826-4f4d-b267-93ecea650c90\" (UID: \"d4c8106e-3826-4f4d-b267-93ecea650c90\") " Oct 30 00:38:25.395897 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:25.395817 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4c8106e-3826-4f4d-b267-93ecea650c90-plugins-config-volume" pod "d4c8106e-3826-4f4d-b267-93ecea650c90" (UID: "d4c8106e-3826-4f4d-b267-93ecea650c90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:38:25.396259 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:25.396237 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4c8106e-3826-4f4d-b267-93ecea650c90-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:25.397845 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:25.397817 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4c8106e-3826-4f4d-b267-93ecea650c90-kube-api-access-ql6cc" pod "d4c8106e-3826-4f4d-b267-93ecea650c90" (UID: "d4c8106e-3826-4f4d-b267-93ecea650c90"). InnerVolumeSpecName "kube-api-access-ql6cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:25.498014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:25.497981 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql6cc\" (UniqueName: \"kubernetes.io/projected/d4c8106e-3826-4f4d-b267-93ecea650c90-kube-api-access-ql6cc\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:25.730287 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:25.730179 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a307fa79-fmcgz" event={"ID":"b4b18928-3805-413b-be9c-7732bc0072a9","Type":"ContainerStarted","Data":"e49b77fdd6dca8bce1b75a05e4eab6b290b76148d2e646c7afd703710a6e6196"} Oct 30 00:38:25.730287 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:25.730246 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a307fa79-fmcgz" event={"ID":"b4b18928-3805-413b-be9c-7732bc0072a9","Type":"ContainerStarted","Data":"4e2f3ade0ab2b9f290ebc623593d714df6364e5a040471bff01bc5b6f3f5346a"} Oct 30 00:38:25.731690 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:25.731666 5001 generic.go:356] "Generic (PLEG): container finished" podID="d4c8106e-3826-4f4d-b267-93ecea650c90" containerID="5145862a61b8d6ac9966ff9c40475f0ae6d733ccdc3404f250ce8b1617fe5821" exitCode=0 Oct 30 00:38:25.731816 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:25.731703 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-29f3ca-6765676c44-vvgqp" Oct 30 00:38:25.731816 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:25.731749 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-29f3ca-6765676c44-vvgqp" event={"ID":"d4c8106e-3826-4f4d-b267-93ecea650c90","Type":"ContainerDied","Data":"5145862a61b8d6ac9966ff9c40475f0ae6d733ccdc3404f250ce8b1617fe5821"} Oct 30 00:38:25.731816 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:25.731795 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-29f3ca-6765676c44-vvgqp" event={"ID":"d4c8106e-3826-4f4d-b267-93ecea650c90","Type":"ContainerDied","Data":"49e61459e072d7664874614527fd35dd32d08d1765b06f8362bb987ee43e3655"} Oct 30 00:38:25.731816 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:25.731816 5001 scope.go:122] "RemoveContainer" containerID="5145862a61b8d6ac9966ff9c40475f0ae6d733ccdc3404f250ce8b1617fe5821" Oct 30 00:38:25.738325 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:25.738297 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc72f0e2-k6fcc" event={"ID":"2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1","Type":"ContainerStarted","Data":"dd8e4ef643d16cafdda70e46b52de32fa2a63062c166e98945107dc1cd53b610"} Oct 30 00:38:25.738442 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:25.738354 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc72f0e2-k6fcc" event={"ID":"2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1","Type":"ContainerStarted","Data":"2c088292e1003ca783d64a82399e13aadfbe845f6bfa23be4758dd28538e0352"} Oct 30 00:38:25.745606 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:25.745588 5001 scope.go:122] "RemoveContainer" containerID="5145862a61b8d6ac9966ff9c40475f0ae6d733ccdc3404f250ce8b1617fe5821" Oct 30 00:38:25.745848 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:38:25.745830 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5145862a61b8d6ac9966ff9c40475f0ae6d733ccdc3404f250ce8b1617fe5821\": container with ID starting with 5145862a61b8d6ac9966ff9c40475f0ae6d733ccdc3404f250ce8b1617fe5821 not found: ID does not exist" containerID="5145862a61b8d6ac9966ff9c40475f0ae6d733ccdc3404f250ce8b1617fe5821" Oct 30 00:38:25.745893 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:25.745857 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5145862a61b8d6ac9966ff9c40475f0ae6d733ccdc3404f250ce8b1617fe5821"} err="failed to get container status \"5145862a61b8d6ac9966ff9c40475f0ae6d733ccdc3404f250ce8b1617fe5821\": rpc error: code = NotFound desc = could not find container \"5145862a61b8d6ac9966ff9c40475f0ae6d733ccdc3404f250ce8b1617fe5821\": container with ID starting with 5145862a61b8d6ac9966ff9c40475f0ae6d733ccdc3404f250ce8b1617fe5821 not found: ID does not exist" Oct 30 00:38:25.768639 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:25.768588 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a307fa79-fmcgz" podStartSLOduration=1.768575984 podStartE2EDuration="1.768575984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:25.764789685 +0000 UTC m=+2020.970290868" watchObservedRunningTime="2026-10-30 00:38:25.768575984 +0000 UTC m=+2020.974077164" Oct 30 00:38:25.802833 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:25.802783 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc72f0e2-k6fcc" podStartSLOduration=1.8027682569999999 podStartE2EDuration="1.802768257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:25.79936896 +0000 UTC m=+2021.004870141" watchObservedRunningTime="2026-10-30 00:38:25.802768257 +0000 UTC m=+2021.008269437" Oct 30 00:38:25.834218 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:25.834187 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-29f3ca-6765676c44-vvgqp"] Oct 30 00:38:25.838730 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:25.838674 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-29f3ca-6765676c44-vvgqp"] Oct 30 00:38:26.293857 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:26.293818 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46"] Oct 30 00:38:26.294131 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:26.294097 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" podUID="ff9c1b8a-7343-4566-9611-7dc87cb6fea5" containerName="istio-proxy" containerID="cri-o://548eb7c34ab72826bbc91fc0d93c7a7ef17404127406a98538ae295715ac3f5f" gracePeriod=30 Oct 30 00:38:26.482028 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:26.476994 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-02665b-c768b5f9b-dpvxb" podUID="a1ebb197-56b8-4043-be23-682f002391b5" containerName="payload-processing" containerID="cri-o://5ae1cf3f2009cca2ebdd16a3ff4d148b0fde0b6a4fdccf05a1a41870fda9b028" gracePeriod=25 Oct 30 00:38:26.749129 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:26.749035 5001 generic.go:356] "Generic (PLEG): container finished" podID="b4b18928-3805-413b-be9c-7732bc0072a9" containerID="e49b77fdd6dca8bce1b75a05e4eab6b290b76148d2e646c7afd703710a6e6196" exitCode=0 Oct 30 00:38:26.749298 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:26.749120 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a307fa79-fmcgz" event={"ID":"b4b18928-3805-413b-be9c-7732bc0072a9","Type":"ContainerDied","Data":"e49b77fdd6dca8bce1b75a05e4eab6b290b76148d2e646c7afd703710a6e6196"} Oct 30 00:38:26.751922 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:26.751881 5001 generic.go:356] "Generic (PLEG): container finished" podID="2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1" containerID="dd8e4ef643d16cafdda70e46b52de32fa2a63062c166e98945107dc1cd53b610" exitCode=0 Oct 30 00:38:26.752059 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:26.751962 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc72f0e2-k6fcc" event={"ID":"2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1","Type":"ContainerDied","Data":"dd8e4ef643d16cafdda70e46b52de32fa2a63062c166e98945107dc1cd53b610"} Oct 30 00:38:26.871746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:26.871709 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-415d24-krpp5"] Oct 30 00:38:26.872418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:26.872400 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d4c8106e-3826-4f4d-b267-93ecea650c90" containerName="payload-processing" Oct 30 00:38:26.872418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:26.872418 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4c8106e-3826-4f4d-b267-93ecea650c90" containerName="payload-processing" Oct 30 00:38:26.872645 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:26.872530 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4c8106e-3826-4f4d-b267-93ecea650c90" containerName="payload-processing" Oct 30 00:38:26.876166 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:26.876144 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-415d24-krpp5" Oct 30 00:38:26.897190 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:26.897163 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-415d24-krpp5"] Oct 30 00:38:26.921819 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:26.921776 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4c8106e-3826-4f4d-b267-93ecea650c90" path="/var/lib/kubelet/pods/d4c8106e-3826-4f4d-b267-93ecea650c90/volumes" Oct 30 00:38:27.027382 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:27.027354 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjrq2\" (UniqueName: \"kubernetes.io/projected/ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1-kube-api-access-rjrq2\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-415d24-krpp5\" (UID: \"ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-415d24-krpp5" Oct 30 00:38:27.027551 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:27.027394 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-415d24-krpp5\" (UID: \"ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-415d24-krpp5" Oct 30 00:38:27.027551 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:27.027487 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-415d24-krpp5\" (UID: \"ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-415d24-krpp5" Oct 30 00:38:27.121171 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:27.121132 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-02665b-c768b5f9b-dpvxb" podUID="a1ebb197-56b8-4043-be23-682f002391b5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:38:27.128741 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:27.128710 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjrq2\" (UniqueName: \"kubernetes.io/projected/ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1-kube-api-access-rjrq2\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-415d24-krpp5\" (UID: \"ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-415d24-krpp5" Oct 30 00:38:27.128892 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:27.128751 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-415d24-krpp5\" (UID: \"ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-415d24-krpp5" Oct 30 00:38:27.128892 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:27.128801 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-415d24-krpp5\" (UID: \"ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-415d24-krpp5" Oct 30 00:38:27.129457 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:27.129436 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-415d24-krpp5\" (UID: \"ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-415d24-krpp5" Oct 30 00:38:27.131271 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:27.131243 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-415d24-krpp5\" (UID: \"ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-415d24-krpp5" Oct 30 00:38:27.149279 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:27.149255 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjrq2\" (UniqueName: \"kubernetes.io/projected/ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1-kube-api-access-rjrq2\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-415d24-krpp5\" (UID: \"ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-415d24-krpp5" Oct 30 00:38:27.194961 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:27.194928 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-415d24-krpp5" Oct 30 00:38:27.499937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:27.499201 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-415d24-krpp5"] Oct 30 00:38:27.763424 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:27.763395 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-415d24-krpp5" event={"ID":"ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1","Type":"ContainerStarted","Data":"2f0cd1d9914e39e0b9360fbe658783fcada5c60c8eee014a6e75284394accf7a"} Oct 30 00:38:27.815792 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:27.815577 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a782e5-7dddfb8f55-tcmx6" podUID="3a97385f-5ca1-41db-9a2a-66cbc70baad2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:38:27.816102 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:27.816071 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a782e5-56b6d8bf96-8pss5" podUID="d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:38:27.904428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:27.904393 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc72f0e2-k6fcc" Oct 30 00:38:27.959729 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:27.959698 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a307fa79-fmcgz" Oct 30 00:38:28.040926 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:28.040829 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4b18928-3805-413b-be9c-7732bc0072a9-kube-api-access-f2jh5\" (UniqueName: \"kubernetes.io/projected/b4b18928-3805-413b-be9c-7732bc0072a9-kube-api-access-f2jh5\") pod \"b4b18928-3805-413b-be9c-7732bc0072a9\" (UID: \"b4b18928-3805-413b-be9c-7732bc0072a9\") " Oct 30 00:38:28.041090 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:28.040945 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1-maas-api-xks-ca\") pod \"2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1\" (UID: \"2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1\") " Oct 30 00:38:28.041090 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:28.040979 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4b18928-3805-413b-be9c-7732bc0072a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4b18928-3805-413b-be9c-7732bc0072a9-maas-api-openshift-ca\") pod \"b4b18928-3805-413b-be9c-7732bc0072a9\" (UID: \"b4b18928-3805-413b-be9c-7732bc0072a9\") " Oct 30 00:38:28.041231 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:28.041086 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1-kube-api-access-fqkw4\" (UniqueName: \"kubernetes.io/projected/2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1-kube-api-access-fqkw4\") pod \"2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1\" (UID: \"2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1\") " Oct 30 00:38:28.041293 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:28.041236 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4b18928-3805-413b-be9c-7732bc0072a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4b18928-3805-413b-be9c-7732bc0072a9-maas-api-xks-ca\") pod \"b4b18928-3805-413b-be9c-7732bc0072a9\" (UID: \"b4b18928-3805-413b-be9c-7732bc0072a9\") " Oct 30 00:38:28.041293 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:28.041286 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1-maas-api-openshift-ca\") pod \"2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1\" (UID: \"2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1\") " Oct 30 00:38:28.041515 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:28.041417 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4b18928-3805-413b-be9c-7732bc0072a9-maas-api-openshift-ca" pod "b4b18928-3805-413b-be9c-7732bc0072a9" (UID: "b4b18928-3805-413b-be9c-7732bc0072a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:38:28.042096 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:28.041873 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4b18928-3805-413b-be9c-7732bc0072a9-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:28.042096 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:28.042037 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1-maas-api-openshift-ca" pod "2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1" (UID: "2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:38:28.043304 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:28.043274 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1-maas-api-xks-ca" pod "2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1" (UID: "2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:38:28.043967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:28.043891 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4b18928-3805-413b-be9c-7732bc0072a9-maas-api-xks-ca" pod "b4b18928-3805-413b-be9c-7732bc0072a9" (UID: "b4b18928-3805-413b-be9c-7732bc0072a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:38:28.044312 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:28.044289 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1-kube-api-access-fqkw4" pod "2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1" (UID: "2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1"). InnerVolumeSpecName "kube-api-access-fqkw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:28.044415 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:28.044329 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4b18928-3805-413b-be9c-7732bc0072a9-kube-api-access-f2jh5" pod "b4b18928-3805-413b-be9c-7732bc0072a9" (UID: "b4b18928-3805-413b-be9c-7732bc0072a9"). InnerVolumeSpecName "kube-api-access-f2jh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:28.143692 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:28.143665 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4b18928-3805-413b-be9c-7732bc0072a9-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:28.143692 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:28.143692 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:28.143903 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:28.143703 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2jh5\" (UniqueName: \"kubernetes.io/projected/b4b18928-3805-413b-be9c-7732bc0072a9-kube-api-access-f2jh5\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:28.143903 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:28.143714 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:28.143903 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:28.143722 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqkw4\" (UniqueName: \"kubernetes.io/projected/2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1-kube-api-access-fqkw4\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:28.775893 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:28.775859 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a307fa79-fmcgz" Oct 30 00:38:28.776401 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:28.775890 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a307fa79-fmcgz" event={"ID":"b4b18928-3805-413b-be9c-7732bc0072a9","Type":"ContainerDied","Data":"4e2f3ade0ab2b9f290ebc623593d714df6364e5a040471bff01bc5b6f3f5346a"} Oct 30 00:38:28.776401 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:28.775981 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e2f3ade0ab2b9f290ebc623593d714df6364e5a040471bff01bc5b6f3f5346a" Oct 30 00:38:28.778596 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:28.778561 5001 generic.go:356] "Generic (PLEG): container finished" podID="ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1" containerID="54322458f8932332ae106912a3b6d3b3b6da99f2e30bef3fb725525cea1740a0" exitCode=0 Oct 30 00:38:28.778773 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:28.778654 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-415d24-krpp5" event={"ID":"ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1","Type":"ContainerDied","Data":"54322458f8932332ae106912a3b6d3b3b6da99f2e30bef3fb725525cea1740a0"} Oct 30 00:38:28.780233 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:28.780214 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc72f0e2-k6fcc" Oct 30 00:38:28.780381 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:28.780252 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc72f0e2-k6fcc" event={"ID":"2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1","Type":"ContainerDied","Data":"2c088292e1003ca783d64a82399e13aadfbe845f6bfa23be4758dd28538e0352"} Oct 30 00:38:28.780381 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:28.780291 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c088292e1003ca783d64a82399e13aadfbe845f6bfa23be4758dd28538e0352" Oct 30 00:38:29.940939 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:29.940894 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-415d24-krpp5" Oct 30 00:38:30.079487 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:30.079323 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1-maas-api-openshift-ca\") pod \"ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1\" (UID: \"ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1\") " Oct 30 00:38:30.079487 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:30.079458 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1-maas-api-xks-ca\") pod \"ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1\" (UID: \"ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1\") " Oct 30 00:38:30.080124 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:30.079502 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1-kube-api-access-rjrq2\" (UniqueName: \"kubernetes.io/projected/ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1-kube-api-access-rjrq2\") pod \"ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1\" (UID: \"ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1\") " Oct 30 00:38:30.082836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:30.081487 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1-maas-api-openshift-ca" pod "ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1" (UID: "ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:38:30.088686 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:30.088638 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1-maas-api-xks-ca" pod "ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1" (UID: "ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:38:30.096341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:30.096292 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1-kube-api-access-rjrq2" pod "ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1" (UID: "ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1"). InnerVolumeSpecName "kube-api-access-rjrq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:30.181579 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:30.181537 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:30.181579 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:30.181574 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:30.181794 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:30.181589 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjrq2\" (UniqueName: \"kubernetes.io/projected/ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1-kube-api-access-rjrq2\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:30.753378 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:30.753335 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" containerID="cri-o://ab6e9faeb5bd0521dc1e733d5ad31678e9a73e15086f6826894c076196c0570f" gracePeriod=45 Oct 30 00:38:30.766127 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:30.766091 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": dial tcp 10.132.0.91:8000: connect: connection refused" Oct 30 00:38:30.800317 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:30.800281 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-415d24-krpp5" event={"ID":"ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1","Type":"ContainerDied","Data":"2f0cd1d9914e39e0b9360fbe658783fcada5c60c8eee014a6e75284394accf7a"} Oct 30 00:38:30.800482 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:30.800369 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f0cd1d9914e39e0b9360fbe658783fcada5c60c8eee014a6e75284394accf7a" Oct 30 00:38:30.800482 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:30.800293 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-415d24-krpp5" Oct 30 00:38:31.028093 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.028069 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:38:31.104391 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.104270 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm"] Oct 30 00:38:31.104888 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.104833 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" podUID="b41b47fa-39f2-4620-b648-797837a6cd1a" containerName="istio-proxy" containerID="cri-o://cbf662eb37a2c8bb40b8c72ae27a2fd53844eecfe68b4d677430646c3591f02b" gracePeriod=30 Oct 30 00:38:31.202574 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.202541 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-kube-api-access-87nwg\" (UniqueName: \"kubernetes.io/projected/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-kube-api-access-87nwg\") pod \"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9\" (UID: \"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9\") " Oct 30 00:38:31.202788 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.202670 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-dshm\") pod \"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9\" (UID: \"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9\") " Oct 30 00:38:31.202788 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.202712 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-tmp-dir\") pod \"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9\" (UID: \"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9\") " Oct 30 00:38:31.202788 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.202756 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-model-cache\") pod \"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9\" (UID: \"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9\") " Oct 30 00:38:31.203008 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.202790 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-tls-certs\" (UniqueName: \"kubernetes.io/secret/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-tls-certs\") pod \"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9\" (UID: \"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9\") " Oct 30 00:38:31.203008 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.202841 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-home\" (UniqueName: \"kubernetes.io/empty-dir/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-home\") pod \"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9\" (UID: \"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9\") " Oct 30 00:38:31.203160 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.203130 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-model-cache" pod "b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" (UID: "b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:31.203383 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.203360 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-home" pod "b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" (UID: "b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:31.203573 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.203555 5001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-model-cache\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:31.203677 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.203578 5001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-home\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:31.203677 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.203556 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-tmp-dir" pod "b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" (UID: "b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:31.205185 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.205156 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-kube-api-access-87nwg" pod "b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" (UID: "b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9"). InnerVolumeSpecName "kube-api-access-87nwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:31.205554 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.205527 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-dshm" pod "b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" (UID: "b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:31.205637 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.205552 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-tls-certs" pod "b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" (UID: "b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:38:31.305551 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.305463 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87nwg\" (UniqueName: \"kubernetes.io/projected/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-kube-api-access-87nwg\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:31.305551 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.305500 5001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-dshm\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:31.305551 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.305537 5001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-tmp-dir\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:31.305551 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.305550 5001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9-tls-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:31.585888 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.585855 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:38:31.676609 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.676577 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-05c6b2-kzxws"] Oct 30 00:38:31.677200 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.677184 5001 cpu_manager.go:414] "containerMap: removing container" podUID="2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1" containerName="revoke-keys" Oct 30 00:38:31.677277 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.677200 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1" containerName="revoke-keys" Oct 30 00:38:31.677277 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.677214 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b4b18928-3805-413b-be9c-7732bc0072a9" containerName="revoke-keys" Oct 30 00:38:31.677277 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.677220 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4b18928-3805-413b-be9c-7732bc0072a9" containerName="revoke-keys" Oct 30 00:38:31.677277 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.677228 5001 cpu_manager.go:414] "containerMap: removing container" podUID="ff9c1b8a-7343-4566-9611-7dc87cb6fea5" containerName="istio-proxy" Oct 30 00:38:31.677277 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.677233 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff9c1b8a-7343-4566-9611-7dc87cb6fea5" containerName="istio-proxy" Oct 30 00:38:31.677277 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.677243 5001 cpu_manager.go:414] "containerMap: removing container" podUID="ff9c1b8a-7343-4566-9611-7dc87cb6fea5" containerName="istio-proxy" Oct 30 00:38:31.677277 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.677249 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff9c1b8a-7343-4566-9611-7dc87cb6fea5" containerName="istio-proxy" Oct 30 00:38:31.677277 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.677259 5001 cpu_manager.go:414] "containerMap: removing container" podUID="ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1" containerName="revoke-keys" Oct 30 00:38:31.677277 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.677263 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1" containerName="revoke-keys" Oct 30 00:38:31.677277 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.677277 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" Oct 30 00:38:31.677277 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.677282 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" Oct 30 00:38:31.677631 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.677368 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1" containerName="revoke-keys" Oct 30 00:38:31.677631 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.677378 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4b18928-3805-413b-be9c-7732bc0072a9" containerName="revoke-keys" Oct 30 00:38:31.677631 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.677384 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff9c1b8a-7343-4566-9611-7dc87cb6fea5" containerName="istio-proxy" Oct 30 00:38:31.677631 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.677391 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1" containerName="revoke-keys" Oct 30 00:38:31.677631 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.677398 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerName="main" Oct 30 00:38:31.680938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.680920 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-05c6b2-kzxws" Oct 30 00:38:31.696469 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.696437 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:38:31.696613 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.696475 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:38:31.696613 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.696475 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:38:31.696613 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.696504 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:38:31.702028 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.701998 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-05c6b2-kzxws"] Oct 30 00:38:31.711015 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.710977 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-credential-socket\") pod \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " Oct 30 00:38:31.711150 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.711114 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istio-envoy\") pod \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " Oct 30 00:38:31.711224 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.711158 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istio-token\" (UniqueName: \"kubernetes.io/projected/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istio-token\") pod \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " Oct 30 00:38:31.711287 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.711219 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istio-data\") pod \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " Oct 30 00:38:31.711287 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.711269 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-workload-socket\") pod \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " Oct 30 00:38:31.711424 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.711298 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istio-podinfo\") pod \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " Oct 30 00:38:31.711424 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.711367 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-workload-certs\") pod \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " Oct 30 00:38:31.711540 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.711422 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istiod-ca-cert\") pod \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " Oct 30 00:38:31.711540 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.711470 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-kube-api-access-69pdb\" (UniqueName: \"kubernetes.io/projected/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-kube-api-access-69pdb\") pod \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\" (UID: \"ff9c1b8a-7343-4566-9611-7dc87cb6fea5\") " Oct 30 00:38:31.712962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.712430 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istio-data" pod "ff9c1b8a-7343-4566-9611-7dc87cb6fea5" (UID: "ff9c1b8a-7343-4566-9611-7dc87cb6fea5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:31.712962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.712646 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-workload-socket" pod "ff9c1b8a-7343-4566-9611-7dc87cb6fea5" (UID: "ff9c1b8a-7343-4566-9611-7dc87cb6fea5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:31.712962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.712865 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-workload-certs" pod "ff9c1b8a-7343-4566-9611-7dc87cb6fea5" (UID: "ff9c1b8a-7343-4566-9611-7dc87cb6fea5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:31.713210 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.713051 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istiod-ca-cert" pod "ff9c1b8a-7343-4566-9611-7dc87cb6fea5" (UID: "ff9c1b8a-7343-4566-9611-7dc87cb6fea5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:38:31.713466 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.713432 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-credential-socket" pod "ff9c1b8a-7343-4566-9611-7dc87cb6fea5" (UID: "ff9c1b8a-7343-4566-9611-7dc87cb6fea5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:31.715067 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.715046 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istio-podinfo" pod "ff9c1b8a-7343-4566-9611-7dc87cb6fea5" (UID: "ff9c1b8a-7343-4566-9611-7dc87cb6fea5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:38:31.715169 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.715131 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-kube-api-access-69pdb" pod "ff9c1b8a-7343-4566-9611-7dc87cb6fea5" (UID: "ff9c1b8a-7343-4566-9611-7dc87cb6fea5"). InnerVolumeSpecName "kube-api-access-69pdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:31.715463 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.715441 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istio-token" pod "ff9c1b8a-7343-4566-9611-7dc87cb6fea5" (UID: "ff9c1b8a-7343-4566-9611-7dc87cb6fea5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:31.716072 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.716054 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istio-envoy" pod "ff9c1b8a-7343-4566-9611-7dc87cb6fea5" (UID: "ff9c1b8a-7343-4566-9611-7dc87cb6fea5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:31.810364 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.810332 5001 generic.go:356] "Generic (PLEG): container finished" podID="ff9c1b8a-7343-4566-9611-7dc87cb6fea5" containerID="548eb7c34ab72826bbc91fc0d93c7a7ef17404127406a98538ae295715ac3f5f" exitCode=0 Oct 30 00:38:31.810527 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.810385 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" Oct 30 00:38:31.810527 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.810415 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" event={"ID":"ff9c1b8a-7343-4566-9611-7dc87cb6fea5","Type":"ContainerDied","Data":"548eb7c34ab72826bbc91fc0d93c7a7ef17404127406a98538ae295715ac3f5f"} Oct 30 00:38:31.810527 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.810473 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46" event={"ID":"ff9c1b8a-7343-4566-9611-7dc87cb6fea5","Type":"ContainerDied","Data":"c68617f2d0b5d648c767265e6c5becc71c97cad8ec20f9ed89f2012c69d6d52e"} Oct 30 00:38:31.810527 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.810504 5001 scope.go:122] "RemoveContainer" containerID="548eb7c34ab72826bbc91fc0d93c7a7ef17404127406a98538ae295715ac3f5f" Oct 30 00:38:31.811977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.811946 5001 generic.go:356] "Generic (PLEG): container finished" podID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" containerID="ab6e9faeb5bd0521dc1e733d5ad31678e9a73e15086f6826894c076196c0570f" exitCode=0 Oct 30 00:38:31.812091 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.811986 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" Oct 30 00:38:31.812091 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.811995 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" event={"ID":"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9","Type":"ContainerDied","Data":"ab6e9faeb5bd0521dc1e733d5ad31678e9a73e15086f6826894c076196c0570f"} Oct 30 00:38:31.812091 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.812032 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s" event={"ID":"b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9","Type":"ContainerDied","Data":"2cd96961a82edf577cfc1af4c168393ce4ae94e5fffd6c50d462b12b847a5b5e"} Oct 30 00:38:31.812665 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.812642 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cf74bb9-4855-4415-ae14-301349aad6d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-05c6b2-kzxws\" (UID: \"5cf74bb9-4855-4415-ae14-301349aad6d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-05c6b2-kzxws" Oct 30 00:38:31.812768 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.812688 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqx2p\" (UniqueName: \"kubernetes.io/projected/5cf74bb9-4855-4415-ae14-301349aad6d3-kube-api-access-pqx2p\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-05c6b2-kzxws\" (UID: \"5cf74bb9-4855-4415-ae14-301349aad6d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-05c6b2-kzxws" Oct 30 00:38:31.812841 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.812812 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cf74bb9-4855-4415-ae14-301349aad6d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-05c6b2-kzxws\" (UID: \"5cf74bb9-4855-4415-ae14-301349aad6d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-05c6b2-kzxws" Oct 30 00:38:31.812972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.812955 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:31.813044 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.812976 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:31.813044 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.812988 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:31.813044 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.813001 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:31.813044 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.813011 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:31.813044 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.813023 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69pdb\" (UniqueName: \"kubernetes.io/projected/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-kube-api-access-69pdb\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:31.813044 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.813035 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:31.813044 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.813044 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:31.813425 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.813052 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff9c1b8a-7343-4566-9611-7dc87cb6fea5-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:31.824171 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.824153 5001 scope.go:122] "RemoveContainer" containerID="43b8ad597cf0aaaad4f4de137cc30ef303022bd09c4c21700397e0ae15d15a30" Oct 30 00:38:31.850945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.847191 5001 scope.go:122] "RemoveContainer" containerID="548eb7c34ab72826bbc91fc0d93c7a7ef17404127406a98538ae295715ac3f5f" Oct 30 00:38:31.850945 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:38:31.850734 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"548eb7c34ab72826bbc91fc0d93c7a7ef17404127406a98538ae295715ac3f5f\": container with ID starting with 548eb7c34ab72826bbc91fc0d93c7a7ef17404127406a98538ae295715ac3f5f not found: ID does not exist" containerID="548eb7c34ab72826bbc91fc0d93c7a7ef17404127406a98538ae295715ac3f5f" Oct 30 00:38:31.850945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.850776 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"548eb7c34ab72826bbc91fc0d93c7a7ef17404127406a98538ae295715ac3f5f"} err="failed to get container status \"548eb7c34ab72826bbc91fc0d93c7a7ef17404127406a98538ae295715ac3f5f\": rpc error: code = NotFound desc = could not find container \"548eb7c34ab72826bbc91fc0d93c7a7ef17404127406a98538ae295715ac3f5f\": container with ID starting with 548eb7c34ab72826bbc91fc0d93c7a7ef17404127406a98538ae295715ac3f5f not found: ID does not exist" Oct 30 00:38:31.850945 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.850803 5001 scope.go:122] "RemoveContainer" containerID="43b8ad597cf0aaaad4f4de137cc30ef303022bd09c4c21700397e0ae15d15a30" Oct 30 00:38:31.854390 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:38:31.851961 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43b8ad597cf0aaaad4f4de137cc30ef303022bd09c4c21700397e0ae15d15a30\": container with ID starting with 43b8ad597cf0aaaad4f4de137cc30ef303022bd09c4c21700397e0ae15d15a30 not found: ID does not exist" containerID="43b8ad597cf0aaaad4f4de137cc30ef303022bd09c4c21700397e0ae15d15a30" Oct 30 00:38:31.854390 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.852007 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43b8ad597cf0aaaad4f4de137cc30ef303022bd09c4c21700397e0ae15d15a30"} err="failed to get container status \"43b8ad597cf0aaaad4f4de137cc30ef303022bd09c4c21700397e0ae15d15a30\": rpc error: code = NotFound desc = could not find container \"43b8ad597cf0aaaad4f4de137cc30ef303022bd09c4c21700397e0ae15d15a30\": container with ID starting with 43b8ad597cf0aaaad4f4de137cc30ef303022bd09c4c21700397e0ae15d15a30 not found: ID does not exist" Oct 30 00:38:31.854390 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.852029 5001 scope.go:122] "RemoveContainer" containerID="ab6e9faeb5bd0521dc1e733d5ad31678e9a73e15086f6826894c076196c0570f" Oct 30 00:38:31.870905 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.870872 5001 scope.go:122] "RemoveContainer" containerID="ab6e9faeb5bd0521dc1e733d5ad31678e9a73e15086f6826894c076196c0570f" Oct 30 00:38:31.871995 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:38:31.871970 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab6e9faeb5bd0521dc1e733d5ad31678e9a73e15086f6826894c076196c0570f\": container with ID starting with ab6e9faeb5bd0521dc1e733d5ad31678e9a73e15086f6826894c076196c0570f not found: ID does not exist" containerID="ab6e9faeb5bd0521dc1e733d5ad31678e9a73e15086f6826894c076196c0570f" Oct 30 00:38:31.872097 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.872004 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab6e9faeb5bd0521dc1e733d5ad31678e9a73e15086f6826894c076196c0570f"} err="failed to get container status \"ab6e9faeb5bd0521dc1e733d5ad31678e9a73e15086f6826894c076196c0570f\": rpc error: code = NotFound desc = could not find container \"ab6e9faeb5bd0521dc1e733d5ad31678e9a73e15086f6826894c076196c0570f\": container with ID starting with ab6e9faeb5bd0521dc1e733d5ad31678e9a73e15086f6826894c076196c0570f not found: ID does not exist" Oct 30 00:38:31.878693 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.878669 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s"] Oct 30 00:38:31.883386 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.883364 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-02665b/distinct-w4-02665b-kserve-84cb745cdd-gbp4s"] Oct 30 00:38:31.911366 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.911336 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46"] Oct 30 00:38:31.914150 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.914123 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cf74bb9-4855-4415-ae14-301349aad6d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-05c6b2-kzxws\" (UID: \"5cf74bb9-4855-4415-ae14-301349aad6d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-05c6b2-kzxws" Oct 30 00:38:31.914317 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.914269 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cf74bb9-4855-4415-ae14-301349aad6d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-05c6b2-kzxws\" (UID: \"5cf74bb9-4855-4415-ae14-301349aad6d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-05c6b2-kzxws" Oct 30 00:38:31.914395 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.914322 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqx2p\" (UniqueName: \"kubernetes.io/projected/5cf74bb9-4855-4415-ae14-301349aad6d3-kube-api-access-pqx2p\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-05c6b2-kzxws\" (UID: \"5cf74bb9-4855-4415-ae14-301349aad6d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-05c6b2-kzxws" Oct 30 00:38:31.915019 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.914999 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cf74bb9-4855-4415-ae14-301349aad6d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-05c6b2-kzxws\" (UID: \"5cf74bb9-4855-4415-ae14-301349aad6d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-05c6b2-kzxws" Oct 30 00:38:31.915811 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.915791 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-29f3ca-openshift-default-7f749db475-hdz46"] Oct 30 00:38:31.916754 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.916730 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cf74bb9-4855-4415-ae14-301349aad6d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-05c6b2-kzxws\" (UID: \"5cf74bb9-4855-4415-ae14-301349aad6d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-05c6b2-kzxws" Oct 30 00:38:31.934770 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.934745 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqx2p\" (UniqueName: \"kubernetes.io/projected/5cf74bb9-4855-4415-ae14-301349aad6d3-kube-api-access-pqx2p\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-05c6b2-kzxws\" (UID: \"5cf74bb9-4855-4415-ae14-301349aad6d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-05c6b2-kzxws" Oct 30 00:38:31.996044 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:31.996013 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-05c6b2-kzxws" Oct 30 00:38:32.303209 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:38:32.303174 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf74bb9_4855_4415_ae14_301349aad6d3.slice/crio-1e616a2535ff32eab9feaf59f4c7f9c8b8828824b49797d977b85a02c6d9cd06 WatchSource:0}: Error finding container 1e616a2535ff32eab9feaf59f4c7f9c8b8828824b49797d977b85a02c6d9cd06: Status 404 returned error can't find the container with id 1e616a2535ff32eab9feaf59f4c7f9c8b8828824b49797d977b85a02c6d9cd06 Oct 30 00:38:32.305159 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:32.305123 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-05c6b2-kzxws"] Oct 30 00:38:32.824414 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:32.824371 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-05c6b2-kzxws" event={"ID":"5cf74bb9-4855-4415-ae14-301349aad6d3","Type":"ContainerStarted","Data":"ab7b1afbd2ceffee7f1bdddb4f84f525e51f343bea12fdcb01a6f8900b9a87c8"} Oct 30 00:38:32.824605 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:32.824438 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-05c6b2-kzxws" event={"ID":"5cf74bb9-4855-4415-ae14-301349aad6d3","Type":"ContainerStarted","Data":"1e616a2535ff32eab9feaf59f4c7f9c8b8828824b49797d977b85a02c6d9cd06"} Oct 30 00:38:32.859742 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:32.859695 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-05c6b2-kzxws" podStartSLOduration=1.859682708 podStartE2EDuration="1.859682708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:32.858980507 +0000 UTC m=+2028.064481689" watchObservedRunningTime="2026-10-30 00:38:32.859682708 +0000 UTC m=+2028.065183888" Oct 30 00:38:32.919136 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:32.919055 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9" path="/var/lib/kubelet/pods/b3fa62b6-7b40-4b35-b3aa-926e1ab8bee9/volumes" Oct 30 00:38:32.919578 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:32.919560 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff9c1b8a-7343-4566-9611-7dc87cb6fea5" path="/var/lib/kubelet/pods/ff9c1b8a-7343-4566-9611-7dc87cb6fea5/volumes" Oct 30 00:38:33.834740 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:33.834706 5001 generic.go:356] "Generic (PLEG): container finished" podID="5cf74bb9-4855-4415-ae14-301349aad6d3" containerID="ab7b1afbd2ceffee7f1bdddb4f84f525e51f343bea12fdcb01a6f8900b9a87c8" exitCode=0 Oct 30 00:38:33.835178 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:33.834793 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-05c6b2-kzxws" event={"ID":"5cf74bb9-4855-4415-ae14-301349aad6d3","Type":"ContainerDied","Data":"ab7b1afbd2ceffee7f1bdddb4f84f525e51f343bea12fdcb01a6f8900b9a87c8"} Oct 30 00:38:34.994371 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:34.994337 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-05c6b2-kzxws" Oct 30 00:38:35.066424 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.066378 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cf74bb9-4855-4415-ae14-301349aad6d3-kube-api-access-pqx2p\" (UniqueName: \"kubernetes.io/projected/5cf74bb9-4855-4415-ae14-301349aad6d3-kube-api-access-pqx2p\") pod \"5cf74bb9-4855-4415-ae14-301349aad6d3\" (UID: \"5cf74bb9-4855-4415-ae14-301349aad6d3\") " Oct 30 00:38:35.066640 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.066608 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cf74bb9-4855-4415-ae14-301349aad6d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cf74bb9-4855-4415-ae14-301349aad6d3-maas-api-openshift-ca\") pod \"5cf74bb9-4855-4415-ae14-301349aad6d3\" (UID: \"5cf74bb9-4855-4415-ae14-301349aad6d3\") " Oct 30 00:38:35.066713 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.066657 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cf74bb9-4855-4415-ae14-301349aad6d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cf74bb9-4855-4415-ae14-301349aad6d3-maas-api-xks-ca\") pod \"5cf74bb9-4855-4415-ae14-301349aad6d3\" (UID: \"5cf74bb9-4855-4415-ae14-301349aad6d3\") " Oct 30 00:38:35.067073 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.067037 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cf74bb9-4855-4415-ae14-301349aad6d3-maas-api-openshift-ca" pod "5cf74bb9-4855-4415-ae14-301349aad6d3" (UID: "5cf74bb9-4855-4415-ae14-301349aad6d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:38:35.067272 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.067247 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cf74bb9-4855-4415-ae14-301349aad6d3-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:35.068982 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.068955 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf74bb9-4855-4415-ae14-301349aad6d3-kube-api-access-pqx2p" pod "5cf74bb9-4855-4415-ae14-301349aad6d3" (UID: "5cf74bb9-4855-4415-ae14-301349aad6d3"). InnerVolumeSpecName "kube-api-access-pqx2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:35.069167 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.069144 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cf74bb9-4855-4415-ae14-301349aad6d3-maas-api-xks-ca" pod "5cf74bb9-4855-4415-ae14-301349aad6d3" (UID: "5cf74bb9-4855-4415-ae14-301349aad6d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:38:35.168311 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.168226 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cf74bb9-4855-4415-ae14-301349aad6d3-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:35.168311 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.168254 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqx2p\" (UniqueName: \"kubernetes.io/projected/5cf74bb9-4855-4415-ae14-301349aad6d3-kube-api-access-pqx2p\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:35.490034 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.489953 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h"] Oct 30 00:38:35.490631 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.490607 5001 cpu_manager.go:414] "containerMap: removing container" podUID="5cf74bb9-4855-4415-ae14-301349aad6d3" containerName="revoke-keys" Oct 30 00:38:35.490631 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.490629 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cf74bb9-4855-4415-ae14-301349aad6d3" containerName="revoke-keys" Oct 30 00:38:35.490865 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.490846 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cf74bb9-4855-4415-ae14-301349aad6d3" containerName="revoke-keys" Oct 30 00:38:35.491034 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.490878 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff9c1b8a-7343-4566-9611-7dc87cb6fea5" containerName="istio-proxy" Oct 30 00:38:35.494873 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.494773 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:38:35.507420 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.507398 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-d96837-openshift-default-dockercfg-djrp4\"" Oct 30 00:38:35.523873 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.523843 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h"] Oct 30 00:38:35.572866 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.572831 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cad4256-1e03-4119-9cbe-4dbe1717600a-istio-podinfo\") pod \"e2e-autores-d96837-openshift-default-7978c98dcc-w445h\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:38:35.573077 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.572882 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v2hj\" (UniqueName: \"kubernetes.io/projected/6cad4256-1e03-4119-9cbe-4dbe1717600a-kube-api-access-8v2hj\") pod \"e2e-autores-d96837-openshift-default-7978c98dcc-w445h\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:38:35.573077 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.573005 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cad4256-1e03-4119-9cbe-4dbe1717600a-istio-token\") pod \"e2e-autores-d96837-openshift-default-7978c98dcc-w445h\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:38:35.573077 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.573039 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-credential-socket\") pod \"e2e-autores-d96837-openshift-default-7978c98dcc-w445h\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:38:35.573223 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.573137 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-workload-socket\") pod \"e2e-autores-d96837-openshift-default-7978c98dcc-w445h\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:38:35.573223 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.573178 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-workload-certs\") pod \"e2e-autores-d96837-openshift-default-7978c98dcc-w445h\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:38:35.573369 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.573351 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-istio-data\") pod \"e2e-autores-d96837-openshift-default-7978c98dcc-w445h\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:38:35.573436 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.573396 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-istio-envoy\") pod \"e2e-autores-d96837-openshift-default-7978c98dcc-w445h\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:38:35.573495 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.573460 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cad4256-1e03-4119-9cbe-4dbe1717600a-istiod-ca-cert\") pod \"e2e-autores-d96837-openshift-default-7978c98dcc-w445h\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:38:35.675184 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.675136 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-workload-socket\") pod \"e2e-autores-d96837-openshift-default-7978c98dcc-w445h\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:38:35.675386 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.675246 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-workload-certs\") pod \"e2e-autores-d96837-openshift-default-7978c98dcc-w445h\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:38:35.675386 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.675378 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-istio-data\") pod \"e2e-autores-d96837-openshift-default-7978c98dcc-w445h\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:38:35.675483 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.675418 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-istio-envoy\") pod \"e2e-autores-d96837-openshift-default-7978c98dcc-w445h\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:38:35.675483 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.675450 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cad4256-1e03-4119-9cbe-4dbe1717600a-istiod-ca-cert\") pod \"e2e-autores-d96837-openshift-default-7978c98dcc-w445h\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:38:35.675598 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.675582 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cad4256-1e03-4119-9cbe-4dbe1717600a-istio-podinfo\") pod \"e2e-autores-d96837-openshift-default-7978c98dcc-w445h\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:38:35.675648 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.675626 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v2hj\" (UniqueName: \"kubernetes.io/projected/6cad4256-1e03-4119-9cbe-4dbe1717600a-kube-api-access-8v2hj\") pod \"e2e-autores-d96837-openshift-default-7978c98dcc-w445h\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:38:35.675700 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.675685 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cad4256-1e03-4119-9cbe-4dbe1717600a-istio-token\") pod \"e2e-autores-d96837-openshift-default-7978c98dcc-w445h\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:38:35.675744 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.675717 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-credential-socket\") pod \"e2e-autores-d96837-openshift-default-7978c98dcc-w445h\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:38:35.676278 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.676251 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-credential-socket\") pod \"e2e-autores-d96837-openshift-default-7978c98dcc-w445h\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:38:35.677432 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.677378 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-workload-certs\") pod \"e2e-autores-d96837-openshift-default-7978c98dcc-w445h\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:38:35.677693 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.677664 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-workload-socket\") pod \"e2e-autores-d96837-openshift-default-7978c98dcc-w445h\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:38:35.677810 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.677795 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-istio-data\") pod \"e2e-autores-d96837-openshift-default-7978c98dcc-w445h\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:38:35.677878 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.677835 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cad4256-1e03-4119-9cbe-4dbe1717600a-istiod-ca-cert\") pod \"e2e-autores-d96837-openshift-default-7978c98dcc-w445h\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:38:35.691242 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.691203 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-istio-envoy\") pod \"e2e-autores-d96837-openshift-default-7978c98dcc-w445h\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:38:35.691773 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.691748 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cad4256-1e03-4119-9cbe-4dbe1717600a-istio-podinfo\") pod \"e2e-autores-d96837-openshift-default-7978c98dcc-w445h\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:38:35.699293 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.699260 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cad4256-1e03-4119-9cbe-4dbe1717600a-istio-token\") pod \"e2e-autores-d96837-openshift-default-7978c98dcc-w445h\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:38:35.699521 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.699392 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v2hj\" (UniqueName: \"kubernetes.io/projected/6cad4256-1e03-4119-9cbe-4dbe1717600a-kube-api-access-8v2hj\") pod \"e2e-autores-d96837-openshift-default-7978c98dcc-w445h\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:38:35.812853 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.812818 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:38:35.855208 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.855177 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-05c6b2-kzxws" Oct 30 00:38:35.855208 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.855200 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-05c6b2-kzxws" event={"ID":"5cf74bb9-4855-4415-ae14-301349aad6d3","Type":"ContainerDied","Data":"1e616a2535ff32eab9feaf59f4c7f9c8b8828824b49797d977b85a02c6d9cd06"} Oct 30 00:38:35.855412 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:35.855249 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e616a2535ff32eab9feaf59f4c7f9c8b8828824b49797d977b85a02c6d9cd06" Oct 30 00:38:36.139506 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:38:36.139479 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cad4256_1e03_4119_9cbe_4dbe1717600a.slice/crio-957b4af3da09d5e5dd01792f0ae4f7eed861623b1286152a755975c01b48a8c9 WatchSource:0}: Error finding container 957b4af3da09d5e5dd01792f0ae4f7eed861623b1286152a755975c01b48a8c9: Status 404 returned error can't find the container with id 957b4af3da09d5e5dd01792f0ae4f7eed861623b1286152a755975c01b48a8c9 Oct 30 00:38:36.142583 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.142553 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h"] Oct 30 00:38:36.217629 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.217576 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:38:36.217778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.217676 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:38:36.217778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.217719 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:38:36.414565 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.414536 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:38:36.492416 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.491153 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-workload-socket\") pod \"b41b47fa-39f2-4620-b648-797837a6cd1a\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " Oct 30 00:38:36.492416 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.491219 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b41b47fa-39f2-4620-b648-797837a6cd1a-kube-api-access-6g9c6\" (UniqueName: \"kubernetes.io/projected/b41b47fa-39f2-4620-b648-797837a6cd1a-kube-api-access-6g9c6\") pod \"b41b47fa-39f2-4620-b648-797837a6cd1a\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " Oct 30 00:38:36.492416 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.491295 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b41b47fa-39f2-4620-b648-797837a6cd1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b41b47fa-39f2-4620-b648-797837a6cd1a-istio-podinfo\") pod \"b41b47fa-39f2-4620-b648-797837a6cd1a\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " Oct 30 00:38:36.492416 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.491380 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-istio-envoy\") pod \"b41b47fa-39f2-4620-b648-797837a6cd1a\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " Oct 30 00:38:36.492416 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.491458 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-istio-data\") pod \"b41b47fa-39f2-4620-b648-797837a6cd1a\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " Oct 30 00:38:36.492416 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.491518 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b41b47fa-39f2-4620-b648-797837a6cd1a-istio-token\" (UniqueName: \"kubernetes.io/projected/b41b47fa-39f2-4620-b648-797837a6cd1a-istio-token\") pod \"b41b47fa-39f2-4620-b648-797837a6cd1a\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " Oct 30 00:38:36.492416 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.491559 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-workload-certs\") pod \"b41b47fa-39f2-4620-b648-797837a6cd1a\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " Oct 30 00:38:36.492416 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.491652 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-credential-socket\") pod \"b41b47fa-39f2-4620-b648-797837a6cd1a\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " Oct 30 00:38:36.492416 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.491731 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b41b47fa-39f2-4620-b648-797837a6cd1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b41b47fa-39f2-4620-b648-797837a6cd1a-istiod-ca-cert\") pod \"b41b47fa-39f2-4620-b648-797837a6cd1a\" (UID: \"b41b47fa-39f2-4620-b648-797837a6cd1a\") " Oct 30 00:38:36.493703 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.493468 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b41b47fa-39f2-4620-b648-797837a6cd1a-istiod-ca-cert" pod "b41b47fa-39f2-4620-b648-797837a6cd1a" (UID: "b41b47fa-39f2-4620-b648-797837a6cd1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:38:36.494166 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.493862 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-istio-data" pod "b41b47fa-39f2-4620-b648-797837a6cd1a" (UID: "b41b47fa-39f2-4620-b648-797837a6cd1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:36.494631 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.494538 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-workload-certs" pod "b41b47fa-39f2-4620-b648-797837a6cd1a" (UID: "b41b47fa-39f2-4620-b648-797837a6cd1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:36.494735 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.494641 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-istio-envoy" pod "b41b47fa-39f2-4620-b648-797837a6cd1a" (UID: "b41b47fa-39f2-4620-b648-797837a6cd1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:36.494735 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.494699 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-workload-socket" pod "b41b47fa-39f2-4620-b648-797837a6cd1a" (UID: "b41b47fa-39f2-4620-b648-797837a6cd1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:36.494873 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.494759 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-credential-socket" pod "b41b47fa-39f2-4620-b648-797837a6cd1a" (UID: "b41b47fa-39f2-4620-b648-797837a6cd1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:36.496492 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.496460 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b41b47fa-39f2-4620-b648-797837a6cd1a-istio-token" pod "b41b47fa-39f2-4620-b648-797837a6cd1a" (UID: "b41b47fa-39f2-4620-b648-797837a6cd1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:36.497350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.497314 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b41b47fa-39f2-4620-b648-797837a6cd1a-istio-podinfo" pod "b41b47fa-39f2-4620-b648-797837a6cd1a" (UID: "b41b47fa-39f2-4620-b648-797837a6cd1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:38:36.497543 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.497518 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b41b47fa-39f2-4620-b648-797837a6cd1a-kube-api-access-6g9c6" pod "b41b47fa-39f2-4620-b648-797837a6cd1a" (UID: "b41b47fa-39f2-4620-b648-797837a6cd1a"). InnerVolumeSpecName "kube-api-access-6g9c6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:36.593753 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.593705 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b41b47fa-39f2-4620-b648-797837a6cd1a-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:36.593753 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.593749 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:36.594029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.593767 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g9c6\" (UniqueName: \"kubernetes.io/projected/b41b47fa-39f2-4620-b648-797837a6cd1a-kube-api-access-6g9c6\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:36.594029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.593782 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b41b47fa-39f2-4620-b648-797837a6cd1a-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:36.594029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.593798 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:36.594029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.593873 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:36.594029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.593887 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b41b47fa-39f2-4620-b648-797837a6cd1a-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:36.594029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.593901 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:36.594029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.593930 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41b47fa-39f2-4620-b648-797837a6cd1a-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:36.920358 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.920315 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" event={"ID":"6cad4256-1e03-4119-9cbe-4dbe1717600a","Type":"ContainerStarted","Data":"5a1ecd4d86d27be3c301d8922fa4238fdd9622d99a7e18606a26bf98e429130b"} Oct 30 00:38:36.920650 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.920627 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" event={"ID":"6cad4256-1e03-4119-9cbe-4dbe1717600a","Type":"ContainerStarted","Data":"957b4af3da09d5e5dd01792f0ae4f7eed861623b1286152a755975c01b48a8c9"} Oct 30 00:38:36.948505 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.945092 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx" podUID="218cceb3-ff29-4eba-bd54-d8ea3885aae2" containerName="main" containerID="cri-o://45c8be50728258ee937214ebd2da0186f9ad3adff412b2ae18e6202e4e6b4e38" gracePeriod=45 Oct 30 00:38:36.968299 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.968264 5001 generic.go:356] "Generic (PLEG): container finished" podID="b41b47fa-39f2-4620-b648-797837a6cd1a" containerID="cbf662eb37a2c8bb40b8c72ae27a2fd53844eecfe68b4d677430646c3591f02b" exitCode=0 Oct 30 00:38:36.977805 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.977771 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" Oct 30 00:38:36.994524 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.994495 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" event={"ID":"b41b47fa-39f2-4620-b648-797837a6cd1a","Type":"ContainerDied","Data":"cbf662eb37a2c8bb40b8c72ae27a2fd53844eecfe68b4d677430646c3591f02b"} Oct 30 00:38:36.994737 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.994719 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm" event={"ID":"b41b47fa-39f2-4620-b648-797837a6cd1a","Type":"ContainerDied","Data":"c6b1b6099625ddc906542d0d0f41f21366e3dbd43cb675195ca7f48cb73ab56e"} Oct 30 00:38:36.994851 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.994788 5001 scope.go:122] "RemoveContainer" containerID="cbf662eb37a2c8bb40b8c72ae27a2fd53844eecfe68b4d677430646c3591f02b" Oct 30 00:38:36.999970 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:36.999893 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" podStartSLOduration=1.999873206 podStartE2EDuration="1.999873206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:36.988699068 +0000 UTC m=+2032.194200251" watchObservedRunningTime="2026-10-30 00:38:36.999873206 +0000 UTC m=+2032.205374389" Oct 30 00:38:37.126465 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:37.126412 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-02665b-c768b5f9b-dpvxb" podUID="a1ebb197-56b8-4043-be23-682f002391b5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:38:37.128263 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:37.128229 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm"] Oct 30 00:38:37.136314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:37.131126 5001 scope.go:122] "RemoveContainer" containerID="c18c9fa6e49b4deaa00bf10f962247c1626d4b0e715d446e8281aaabd6755cfe" Oct 30 00:38:37.136314 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:37.133214 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a782e5-openshift-default-77556b6695-6ghhm"] Oct 30 00:38:37.148207 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:37.148181 5001 scope.go:122] "RemoveContainer" containerID="cbf662eb37a2c8bb40b8c72ae27a2fd53844eecfe68b4d677430646c3591f02b" Oct 30 00:38:37.148574 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:38:37.148550 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbf662eb37a2c8bb40b8c72ae27a2fd53844eecfe68b4d677430646c3591f02b\": container with ID starting with cbf662eb37a2c8bb40b8c72ae27a2fd53844eecfe68b4d677430646c3591f02b not found: ID does not exist" containerID="cbf662eb37a2c8bb40b8c72ae27a2fd53844eecfe68b4d677430646c3591f02b" Oct 30 00:38:37.148629 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:37.148587 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf662eb37a2c8bb40b8c72ae27a2fd53844eecfe68b4d677430646c3591f02b"} err="failed to get container status \"cbf662eb37a2c8bb40b8c72ae27a2fd53844eecfe68b4d677430646c3591f02b\": rpc error: code = NotFound desc = could not find container \"cbf662eb37a2c8bb40b8c72ae27a2fd53844eecfe68b4d677430646c3591f02b\": container with ID starting with cbf662eb37a2c8bb40b8c72ae27a2fd53844eecfe68b4d677430646c3591f02b not found: ID does not exist" Oct 30 00:38:37.148629 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:37.148613 5001 scope.go:122] "RemoveContainer" containerID="c18c9fa6e49b4deaa00bf10f962247c1626d4b0e715d446e8281aaabd6755cfe" Oct 30 00:38:37.149409 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:38:37.148905 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c18c9fa6e49b4deaa00bf10f962247c1626d4b0e715d446e8281aaabd6755cfe\": container with ID starting with c18c9fa6e49b4deaa00bf10f962247c1626d4b0e715d446e8281aaabd6755cfe not found: ID does not exist" containerID="c18c9fa6e49b4deaa00bf10f962247c1626d4b0e715d446e8281aaabd6755cfe" Oct 30 00:38:37.149510 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:37.149420 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c18c9fa6e49b4deaa00bf10f962247c1626d4b0e715d446e8281aaabd6755cfe"} err="failed to get container status \"c18c9fa6e49b4deaa00bf10f962247c1626d4b0e715d446e8281aaabd6755cfe\": rpc error: code = NotFound desc = could not find container \"c18c9fa6e49b4deaa00bf10f962247c1626d4b0e715d446e8281aaabd6755cfe\": container with ID starting with c18c9fa6e49b4deaa00bf10f962247c1626d4b0e715d446e8281aaabd6755cfe not found: ID does not exist" Oct 30 00:38:37.295743 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:37.295717 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx" Oct 30 00:38:37.306354 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:37.306324 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/218cceb3-ff29-4eba-bd54-d8ea3885aae2-home\" (UniqueName: \"kubernetes.io/empty-dir/218cceb3-ff29-4eba-bd54-d8ea3885aae2-home\") pod \"218cceb3-ff29-4eba-bd54-d8ea3885aae2\" (UID: \"218cceb3-ff29-4eba-bd54-d8ea3885aae2\") " Oct 30 00:38:37.306488 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:37.306459 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/218cceb3-ff29-4eba-bd54-d8ea3885aae2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/218cceb3-ff29-4eba-bd54-d8ea3885aae2-model-cache\") pod \"218cceb3-ff29-4eba-bd54-d8ea3885aae2\" (UID: \"218cceb3-ff29-4eba-bd54-d8ea3885aae2\") " Oct 30 00:38:37.306584 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:37.306563 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/218cceb3-ff29-4eba-bd54-d8ea3885aae2-home" pod "218cceb3-ff29-4eba-bd54-d8ea3885aae2" (UID: "218cceb3-ff29-4eba-bd54-d8ea3885aae2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:37.306630 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:37.306574 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/218cceb3-ff29-4eba-bd54-d8ea3885aae2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/218cceb3-ff29-4eba-bd54-d8ea3885aae2-dshm\") pod \"218cceb3-ff29-4eba-bd54-d8ea3885aae2\" (UID: \"218cceb3-ff29-4eba-bd54-d8ea3885aae2\") " Oct 30 00:38:37.306678 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:37.306649 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/218cceb3-ff29-4eba-bd54-d8ea3885aae2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/218cceb3-ff29-4eba-bd54-d8ea3885aae2-tmp-dir\") pod \"218cceb3-ff29-4eba-bd54-d8ea3885aae2\" (UID: \"218cceb3-ff29-4eba-bd54-d8ea3885aae2\") " Oct 30 00:38:37.306678 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:37.306675 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/218cceb3-ff29-4eba-bd54-d8ea3885aae2-tls-certs\" (UniqueName: \"kubernetes.io/secret/218cceb3-ff29-4eba-bd54-d8ea3885aae2-tls-certs\") pod \"218cceb3-ff29-4eba-bd54-d8ea3885aae2\" (UID: \"218cceb3-ff29-4eba-bd54-d8ea3885aae2\") " Oct 30 00:38:37.306766 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:37.306740 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/218cceb3-ff29-4eba-bd54-d8ea3885aae2-kube-api-access-2xvqp\" (UniqueName: \"kubernetes.io/projected/218cceb3-ff29-4eba-bd54-d8ea3885aae2-kube-api-access-2xvqp\") pod \"218cceb3-ff29-4eba-bd54-d8ea3885aae2\" (UID: \"218cceb3-ff29-4eba-bd54-d8ea3885aae2\") " Oct 30 00:38:37.306834 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:37.306737 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/218cceb3-ff29-4eba-bd54-d8ea3885aae2-model-cache" pod "218cceb3-ff29-4eba-bd54-d8ea3885aae2" (UID: "218cceb3-ff29-4eba-bd54-d8ea3885aae2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:37.306992 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:37.306969 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/218cceb3-ff29-4eba-bd54-d8ea3885aae2-tmp-dir" pod "218cceb3-ff29-4eba-bd54-d8ea3885aae2" (UID: "218cceb3-ff29-4eba-bd54-d8ea3885aae2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:37.307317 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:37.307265 5001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/218cceb3-ff29-4eba-bd54-d8ea3885aae2-model-cache\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:37.307317 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:37.307289 5001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/218cceb3-ff29-4eba-bd54-d8ea3885aae2-tmp-dir\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:37.307317 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:37.307297 5001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/218cceb3-ff29-4eba-bd54-d8ea3885aae2-home\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:37.309040 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:37.309014 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/218cceb3-ff29-4eba-bd54-d8ea3885aae2-dshm" pod "218cceb3-ff29-4eba-bd54-d8ea3885aae2" (UID: "218cceb3-ff29-4eba-bd54-d8ea3885aae2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:37.309128 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:37.309082 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/218cceb3-ff29-4eba-bd54-d8ea3885aae2-tls-certs" pod "218cceb3-ff29-4eba-bd54-d8ea3885aae2" (UID: "218cceb3-ff29-4eba-bd54-d8ea3885aae2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:38:37.309231 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:37.309207 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/218cceb3-ff29-4eba-bd54-d8ea3885aae2-kube-api-access-2xvqp" pod "218cceb3-ff29-4eba-bd54-d8ea3885aae2" (UID: "218cceb3-ff29-4eba-bd54-d8ea3885aae2"). InnerVolumeSpecName "kube-api-access-2xvqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:37.353990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:37.353942 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-415d24-9fd8c87c8-wn6vg"] Oct 30 00:38:37.380393 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:37.380358 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-415d24-8cc6f76cf-vnt5d"] Oct 30 00:38:37.408647 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:37.408613 5001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/218cceb3-ff29-4eba-bd54-d8ea3885aae2-dshm\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:37.408647 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:37.408642 5001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/218cceb3-ff29-4eba-bd54-d8ea3885aae2-tls-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:37.408647 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:37.408653 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xvqp\" (UniqueName: \"kubernetes.io/projected/218cceb3-ff29-4eba-bd54-d8ea3885aae2-kube-api-access-2xvqp\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:37.813236 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:37.813202 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:38:37.814266 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:37.814237 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a782e5-7dddfb8f55-tcmx6" podUID="3a97385f-5ca1-41db-9a2a-66cbc70baad2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:38:37.814772 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:37.814743 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a782e5-56b6d8bf96-8pss5" podUID="d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:38:37.818481 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:37.818463 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:38:38.006930 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:38.006834 5001 generic.go:356] "Generic (PLEG): container finished" podID="218cceb3-ff29-4eba-bd54-d8ea3885aae2" containerID="45c8be50728258ee937214ebd2da0186f9ad3adff412b2ae18e6202e4e6b4e38" exitCode=0 Oct 30 00:38:38.015967 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:38.008376 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx" Oct 30 00:38:38.020350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:38.017320 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx" event={"ID":"218cceb3-ff29-4eba-bd54-d8ea3885aae2","Type":"ContainerDied","Data":"45c8be50728258ee937214ebd2da0186f9ad3adff412b2ae18e6202e4e6b4e38"} Oct 30 00:38:38.020350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:38.017377 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx" event={"ID":"218cceb3-ff29-4eba-bd54-d8ea3885aae2","Type":"ContainerDied","Data":"cbf1de0687f80fa55409c4234be1fff81adacea8d4512186ffff8fa55d795d92"} Oct 30 00:38:38.020350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:38.017402 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:38:38.020350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:38.017541 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:38:38.020350 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:38.017590 5001 scope.go:122] "RemoveContainer" containerID="45c8be50728258ee937214ebd2da0186f9ad3adff412b2ae18e6202e4e6b4e38" Oct 30 00:38:38.162161 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:38.159902 5001 scope.go:122] "RemoveContainer" containerID="45c8be50728258ee937214ebd2da0186f9ad3adff412b2ae18e6202e4e6b4e38" Oct 30 00:38:38.162161 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:38:38.160323 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45c8be50728258ee937214ebd2da0186f9ad3adff412b2ae18e6202e4e6b4e38\": container with ID starting with 45c8be50728258ee937214ebd2da0186f9ad3adff412b2ae18e6202e4e6b4e38 not found: ID does not exist" containerID="45c8be50728258ee937214ebd2da0186f9ad3adff412b2ae18e6202e4e6b4e38" Oct 30 00:38:38.162161 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:38.160357 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45c8be50728258ee937214ebd2da0186f9ad3adff412b2ae18e6202e4e6b4e38"} err="failed to get container status \"45c8be50728258ee937214ebd2da0186f9ad3adff412b2ae18e6202e4e6b4e38\": rpc error: code = NotFound desc = could not find container \"45c8be50728258ee937214ebd2da0186f9ad3adff412b2ae18e6202e4e6b4e38\": container with ID starting with 45c8be50728258ee937214ebd2da0186f9ad3adff412b2ae18e6202e4e6b4e38 not found: ID does not exist" Oct 30 00:38:38.170180 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:38.170146 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx"] Oct 30 00:38:38.177341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:38.177311 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-63e288/auth-test-model-63e288-kserve-79fc495b49-7s6rx"] Oct 30 00:38:38.863688 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:38.863653 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-02665b-hzfvt"] Oct 30 00:38:38.867778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:38.867754 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-02665b-hzfvt"] Oct 30 00:38:38.936026 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:38.935982 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="218cceb3-ff29-4eba-bd54-d8ea3885aae2" path="/var/lib/kubelet/pods/218cceb3-ff29-4eba-bd54-d8ea3885aae2/volumes" Oct 30 00:38:38.936714 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:38.936670 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d32d4d5-a424-48e4-94aa-fcf81482ebef" path="/var/lib/kubelet/pods/8d32d4d5-a424-48e4-94aa-fcf81482ebef/volumes" Oct 30 00:38:38.937421 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:38.937399 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b41b47fa-39f2-4620-b648-797837a6cd1a" path="/var/lib/kubelet/pods/b41b47fa-39f2-4620-b648-797837a6cd1a/volumes" Oct 30 00:38:41.161950 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:41.161907 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d96837-55b85d6d8c-tgmp9"] Oct 30 00:38:41.162482 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:41.162467 5001 cpu_manager.go:414] "containerMap: removing container" podUID="218cceb3-ff29-4eba-bd54-d8ea3885aae2" containerName="main" Oct 30 00:38:41.162482 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:41.162482 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="218cceb3-ff29-4eba-bd54-d8ea3885aae2" containerName="main" Oct 30 00:38:41.162558 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:41.162502 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b41b47fa-39f2-4620-b648-797837a6cd1a" containerName="istio-proxy" Oct 30 00:38:41.162558 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:41.162507 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b41b47fa-39f2-4620-b648-797837a6cd1a" containerName="istio-proxy" Oct 30 00:38:41.162558 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:41.162515 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b41b47fa-39f2-4620-b648-797837a6cd1a" containerName="istio-proxy" Oct 30 00:38:41.162558 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:41.162519 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b41b47fa-39f2-4620-b648-797837a6cd1a" containerName="istio-proxy" Oct 30 00:38:41.162696 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:41.162630 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="218cceb3-ff29-4eba-bd54-d8ea3885aae2" containerName="main" Oct 30 00:38:41.162696 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:41.162639 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b41b47fa-39f2-4620-b648-797837a6cd1a" containerName="istio-proxy" Oct 30 00:38:41.162696 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:41.162645 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b41b47fa-39f2-4620-b648-797837a6cd1a" containerName="istio-proxy" Oct 30 00:38:41.167290 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:41.167270 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-d96837-55b85d6d8c-tgmp9" Oct 30 00:38:41.181421 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:41.181381 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-d96837\"" Oct 30 00:38:41.186690 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:41.186666 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d96837-55b85d6d8c-tgmp9"] Oct 30 00:38:41.206809 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:41.205576 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-d96837-dockercfg-nxccz\"" Oct 30 00:38:41.328940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:41.300388 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w22gq\" (UniqueName: \"kubernetes.io/projected/0d81642d-32eb-45e9-816b-dee60db16716-kube-api-access-w22gq\") pod \"payload-pre-processing-e2e-autores-d96837-55b85d6d8c-tgmp9\" (UID: \"0d81642d-32eb-45e9-816b-dee60db16716\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d96837-55b85d6d8c-tgmp9" Oct 30 00:38:41.328940 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:41.300508 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d81642d-32eb-45e9-816b-dee60db16716-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-d96837-55b85d6d8c-tgmp9\" (UID: \"0d81642d-32eb-45e9-816b-dee60db16716\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d96837-55b85d6d8c-tgmp9" Oct 30 00:38:41.332391 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:41.332362 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-d96837-5565b8fc78-qwjl5"] Oct 30 00:38:41.344823 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:41.344795 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-d96837-5565b8fc78-qwjl5" Oct 30 00:38:41.363229 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:41.363183 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-d96837-5565b8fc78-qwjl5"] Oct 30 00:38:41.401524 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:41.401482 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd129c0a-51ff-4cd3-85b9-25c05b886998-plugins-config-volume\") pod \"payload-processing-e2e-autores-d96837-5565b8fc78-qwjl5\" (UID: \"cd129c0a-51ff-4cd3-85b9-25c05b886998\") " pod="openshift-ingress/payload-processing-e2e-autores-d96837-5565b8fc78-qwjl5" Oct 30 00:38:41.401696 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:41.401593 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sk88\" (UniqueName: \"kubernetes.io/projected/cd129c0a-51ff-4cd3-85b9-25c05b886998-kube-api-access-7sk88\") pod \"payload-processing-e2e-autores-d96837-5565b8fc78-qwjl5\" (UID: \"cd129c0a-51ff-4cd3-85b9-25c05b886998\") " pod="openshift-ingress/payload-processing-e2e-autores-d96837-5565b8fc78-qwjl5" Oct 30 00:38:41.401696 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:41.401636 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w22gq\" (UniqueName: \"kubernetes.io/projected/0d81642d-32eb-45e9-816b-dee60db16716-kube-api-access-w22gq\") pod \"payload-pre-processing-e2e-autores-d96837-55b85d6d8c-tgmp9\" (UID: \"0d81642d-32eb-45e9-816b-dee60db16716\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d96837-55b85d6d8c-tgmp9" Oct 30 00:38:41.401862 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:41.401706 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d81642d-32eb-45e9-816b-dee60db16716-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-d96837-55b85d6d8c-tgmp9\" (UID: \"0d81642d-32eb-45e9-816b-dee60db16716\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d96837-55b85d6d8c-tgmp9" Oct 30 00:38:41.402445 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:41.402422 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d81642d-32eb-45e9-816b-dee60db16716-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-d96837-55b85d6d8c-tgmp9\" (UID: \"0d81642d-32eb-45e9-816b-dee60db16716\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d96837-55b85d6d8c-tgmp9" Oct 30 00:38:41.422723 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:41.422659 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w22gq\" (UniqueName: \"kubernetes.io/projected/0d81642d-32eb-45e9-816b-dee60db16716-kube-api-access-w22gq\") pod \"payload-pre-processing-e2e-autores-d96837-55b85d6d8c-tgmp9\" (UID: \"0d81642d-32eb-45e9-816b-dee60db16716\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d96837-55b85d6d8c-tgmp9" Oct 30 00:38:41.481731 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:41.481693 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-d96837-55b85d6d8c-tgmp9" Oct 30 00:38:41.502961 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:41.502890 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd129c0a-51ff-4cd3-85b9-25c05b886998-plugins-config-volume\") pod \"payload-processing-e2e-autores-d96837-5565b8fc78-qwjl5\" (UID: \"cd129c0a-51ff-4cd3-85b9-25c05b886998\") " pod="openshift-ingress/payload-processing-e2e-autores-d96837-5565b8fc78-qwjl5" Oct 30 00:38:41.503077 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:41.503046 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sk88\" (UniqueName: \"kubernetes.io/projected/cd129c0a-51ff-4cd3-85b9-25c05b886998-kube-api-access-7sk88\") pod \"payload-processing-e2e-autores-d96837-5565b8fc78-qwjl5\" (UID: \"cd129c0a-51ff-4cd3-85b9-25c05b886998\") " pod="openshift-ingress/payload-processing-e2e-autores-d96837-5565b8fc78-qwjl5" Oct 30 00:38:41.503648 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:41.503629 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd129c0a-51ff-4cd3-85b9-25c05b886998-plugins-config-volume\") pod \"payload-processing-e2e-autores-d96837-5565b8fc78-qwjl5\" (UID: \"cd129c0a-51ff-4cd3-85b9-25c05b886998\") " pod="openshift-ingress/payload-processing-e2e-autores-d96837-5565b8fc78-qwjl5" Oct 30 00:38:41.523805 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:41.523777 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sk88\" (UniqueName: \"kubernetes.io/projected/cd129c0a-51ff-4cd3-85b9-25c05b886998-kube-api-access-7sk88\") pod \"payload-processing-e2e-autores-d96837-5565b8fc78-qwjl5\" (UID: \"cd129c0a-51ff-4cd3-85b9-25c05b886998\") " pod="openshift-ingress/payload-processing-e2e-autores-d96837-5565b8fc78-qwjl5" Oct 30 00:38:41.684431 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:41.684339 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-d96837-5565b8fc78-qwjl5" Oct 30 00:38:41.791323 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:38:41.791289 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d81642d_32eb_45e9_816b_dee60db16716.slice/crio-d312a2cdef32a8840474ea2789439a6ee2fa63c664763280de0b465524a90f96 WatchSource:0}: Error finding container d312a2cdef32a8840474ea2789439a6ee2fa63c664763280de0b465524a90f96: Status 404 returned error can't find the container with id d312a2cdef32a8840474ea2789439a6ee2fa63c664763280de0b465524a90f96 Oct 30 00:38:41.792357 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:41.792299 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d96837-55b85d6d8c-tgmp9"] Oct 30 00:38:41.993334 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:41.993215 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-d96837-5565b8fc78-qwjl5"] Oct 30 00:38:42.046077 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:42.046016 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-d96837-5565b8fc78-qwjl5" event={"ID":"cd129c0a-51ff-4cd3-85b9-25c05b886998","Type":"ContainerStarted","Data":"afaa5db926293590930bc44908d66375f070451362a032e95c48fd661137551f"} Oct 30 00:38:42.047800 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:42.047771 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-d96837-55b85d6d8c-tgmp9" event={"ID":"0d81642d-32eb-45e9-816b-dee60db16716","Type":"ContainerStarted","Data":"8430c9b9da857b6f4cdfcd100205f7b38b7c6ad6c610d86105c82b9773d03fe2"} Oct 30 00:38:42.048655 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:42.047822 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-d96837-55b85d6d8c-tgmp9" event={"ID":"0d81642d-32eb-45e9-816b-dee60db16716","Type":"ContainerStarted","Data":"d312a2cdef32a8840474ea2789439a6ee2fa63c664763280de0b465524a90f96"} Oct 30 00:38:42.049028 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:42.048883 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-d96837-55b85d6d8c-tgmp9" Oct 30 00:38:42.052014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:42.051988 5001 generic.go:356] "Generic (PLEG): container finished" podID="3a97385f-5ca1-41db-9a2a-66cbc70baad2" containerID="4edf7887e39c5427a700c76a4d8230032c53349f5532230dd1762d3e302bd451" exitCode=0 Oct 30 00:38:42.052144 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:42.052085 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a782e5-7dddfb8f55-tcmx6" event={"ID":"3a97385f-5ca1-41db-9a2a-66cbc70baad2","Type":"ContainerDied","Data":"4edf7887e39c5427a700c76a4d8230032c53349f5532230dd1762d3e302bd451"} Oct 30 00:38:42.056026 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:42.055998 5001 generic.go:356] "Generic (PLEG): container finished" podID="d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea" containerID="a687cb927a21b9784bf2a4f2d63fbd9a1ca9aea48c31f4791fca9ac3041fa850" exitCode=0 Oct 30 00:38:42.056124 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:42.056022 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a782e5-56b6d8bf96-8pss5" event={"ID":"d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea","Type":"ContainerDied","Data":"a687cb927a21b9784bf2a4f2d63fbd9a1ca9aea48c31f4791fca9ac3041fa850"} Oct 30 00:38:42.080596 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:42.080566 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a782e5-56b6d8bf96-8pss5" Oct 30 00:38:42.088659 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:42.088605 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-d96837-55b85d6d8c-tgmp9" podStartSLOduration=1.088586102 podStartE2EDuration="1.088586102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:42.083556707 +0000 UTC m=+2037.289057915" watchObservedRunningTime="2026-10-30 00:38:42.088586102 +0000 UTC m=+2037.294087284" Oct 30 00:38:42.108593 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:42.107830 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea-plugins-config-volume\") pod \"d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea\" (UID: \"d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea\") " Oct 30 00:38:42.108593 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:42.108084 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea-kube-api-access-4w8gw\" (UniqueName: \"kubernetes.io/projected/d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea-kube-api-access-4w8gw\") pod \"d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea\" (UID: \"d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea\") " Oct 30 00:38:42.108593 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:42.108174 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea-plugins-config-volume" pod "d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea" (UID: "d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:38:42.108593 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:42.108566 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:42.110735 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:42.110703 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea-kube-api-access-4w8gw" pod "d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea" (UID: "d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea"). InnerVolumeSpecName "kube-api-access-4w8gw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:42.149282 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:42.149255 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a782e5-7dddfb8f55-tcmx6" Oct 30 00:38:42.217864 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:42.217762 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a97385f-5ca1-41db-9a2a-66cbc70baad2-kube-api-access-swtsf\" (UniqueName: \"kubernetes.io/projected/3a97385f-5ca1-41db-9a2a-66cbc70baad2-kube-api-access-swtsf\") pod \"3a97385f-5ca1-41db-9a2a-66cbc70baad2\" (UID: \"3a97385f-5ca1-41db-9a2a-66cbc70baad2\") " Oct 30 00:38:42.218501 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:42.218165 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a97385f-5ca1-41db-9a2a-66cbc70baad2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a97385f-5ca1-41db-9a2a-66cbc70baad2-plugins-config-volume\") pod \"3a97385f-5ca1-41db-9a2a-66cbc70baad2\" (UID: \"3a97385f-5ca1-41db-9a2a-66cbc70baad2\") " Oct 30 00:38:42.218999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:42.218975 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w8gw\" (UniqueName: \"kubernetes.io/projected/d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea-kube-api-access-4w8gw\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:42.219362 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:42.219338 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a97385f-5ca1-41db-9a2a-66cbc70baad2-plugins-config-volume" pod "3a97385f-5ca1-41db-9a2a-66cbc70baad2" (UID: "3a97385f-5ca1-41db-9a2a-66cbc70baad2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:38:42.223553 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:42.223512 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a97385f-5ca1-41db-9a2a-66cbc70baad2-kube-api-access-swtsf" pod "3a97385f-5ca1-41db-9a2a-66cbc70baad2" (UID: "3a97385f-5ca1-41db-9a2a-66cbc70baad2"). InnerVolumeSpecName "kube-api-access-swtsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:42.243698 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:42.243662 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-05c6b2-6f8558fd8b-kslr7"] Oct 30 00:38:42.249576 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:42.249546 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-05c6b2-55b55d747-dgrp9"] Oct 30 00:38:42.320961 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:42.320930 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a97385f-5ca1-41db-9a2a-66cbc70baad2-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:42.321189 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:42.321173 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swtsf\" (UniqueName: \"kubernetes.io/projected/3a97385f-5ca1-41db-9a2a-66cbc70baad2-kube-api-access-swtsf\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:42.356667 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:42.354769 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-415d24-9fd8c87c8-wn6vg" podUID="3f3b775e-8102-4e77-a2f7-272ac5d6ad5a" containerName="payload-processing" containerID="cri-o://5910f3d6ff3668603ced6ac7197ee3fad977de116c28dbcc9f2fc1a6d5f13dce" gracePeriod=25 Oct 30 00:38:42.381652 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:42.381616 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-415d24-8cc6f76cf-vnt5d" podUID="be687896-3ef5-4bfc-b170-3e91fcc13a3e" containerName="payload-pre-processing" containerID="cri-o://84298f705f2b88ee0b66b590765433bfb83e3b2f4996c5e0600fa5a98f2ff830" gracePeriod=25 Oct 30 00:38:43.066324 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:43.066292 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a782e5-56b6d8bf96-8pss5" Oct 30 00:38:43.066324 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:43.066312 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a782e5-56b6d8bf96-8pss5" event={"ID":"d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea","Type":"ContainerDied","Data":"eac22097e50d9b351e35146b5707bb25da967ad3678dcf82aed32bd32d6d1350"} Oct 30 00:38:43.066591 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:43.066377 5001 scope.go:122] "RemoveContainer" containerID="a687cb927a21b9784bf2a4f2d63fbd9a1ca9aea48c31f4791fca9ac3041fa850" Oct 30 00:38:43.068433 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:43.068404 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-d96837-5565b8fc78-qwjl5" event={"ID":"cd129c0a-51ff-4cd3-85b9-25c05b886998","Type":"ContainerStarted","Data":"2ebd72563cb6c7379586440b60469dd1f022b6d8d78aba16e1b00ed4f9c09af1"} Oct 30 00:38:43.069088 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:43.069064 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-d96837-5565b8fc78-qwjl5" Oct 30 00:38:43.070255 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:43.070229 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a782e5-7dddfb8f55-tcmx6" event={"ID":"3a97385f-5ca1-41db-9a2a-66cbc70baad2","Type":"ContainerDied","Data":"e9d8a19cb82fb79d426f4528448c9af95d9d645b7394fb8e390c3ddc231866d8"} Oct 30 00:38:43.070348 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:43.070281 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a782e5-7dddfb8f55-tcmx6" Oct 30 00:38:43.078749 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:43.078728 5001 scope.go:122] "RemoveContainer" containerID="4edf7887e39c5427a700c76a4d8230032c53349f5532230dd1762d3e302bd451" Oct 30 00:38:43.175141 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:43.175073 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-d96837-5565b8fc78-qwjl5" podStartSLOduration=2.175050445 podStartE2EDuration="2.175050445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:43.142362005 +0000 UTC m=+2038.347863188" watchObservedRunningTime="2026-10-30 00:38:43.175050445 +0000 UTC m=+2038.380551627" Oct 30 00:38:43.219069 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:43.218971 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a782e5-7dddfb8f55-tcmx6"] Oct 30 00:38:43.228066 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:43.228035 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a782e5-7dddfb8f55-tcmx6"] Oct 30 00:38:43.241508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:43.241472 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a782e5-56b6d8bf96-8pss5"] Oct 30 00:38:43.247029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:43.246995 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a782e5-56b6d8bf96-8pss5"] Oct 30 00:38:44.919353 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:44.919315 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a97385f-5ca1-41db-9a2a-66cbc70baad2" path="/var/lib/kubelet/pods/3a97385f-5ca1-41db-9a2a-66cbc70baad2/volumes" Oct 30 00:38:44.919802 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:44.919788 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea" path="/var/lib/kubelet/pods/d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea/volumes" Oct 30 00:38:45.977978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:45.977940 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk"] Oct 30 00:38:45.978844 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:45.978792 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" podUID="e723059f-3ca0-45b1-a035-6ce96939fe4b" containerName="istio-proxy" containerID="cri-o://3afb7dc28f6285c6b85072f7c8d2474f1a30607fc0a6fca1ec087c9231579375" gracePeriod=30 Oct 30 00:38:46.663968 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:46.663933 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-02665b-c768b5f9b-dpvxb" Oct 30 00:38:46.696371 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:46.696341 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1ebb197-56b8-4043-be23-682f002391b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ebb197-56b8-4043-be23-682f002391b5-plugins-config-volume\") pod \"a1ebb197-56b8-4043-be23-682f002391b5\" (UID: \"a1ebb197-56b8-4043-be23-682f002391b5\") " Oct 30 00:38:46.696534 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:46.696414 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1ebb197-56b8-4043-be23-682f002391b5-kube-api-access-t9pc9\" (UniqueName: \"kubernetes.io/projected/a1ebb197-56b8-4043-be23-682f002391b5-kube-api-access-t9pc9\") pod \"a1ebb197-56b8-4043-be23-682f002391b5\" (UID: \"a1ebb197-56b8-4043-be23-682f002391b5\") " Oct 30 00:38:46.696789 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:46.696763 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ebb197-56b8-4043-be23-682f002391b5-plugins-config-volume" pod "a1ebb197-56b8-4043-be23-682f002391b5" (UID: "a1ebb197-56b8-4043-be23-682f002391b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:38:46.698654 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:46.698629 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ebb197-56b8-4043-be23-682f002391b5-kube-api-access-t9pc9" pod "a1ebb197-56b8-4043-be23-682f002391b5" (UID: "a1ebb197-56b8-4043-be23-682f002391b5"). InnerVolumeSpecName "kube-api-access-t9pc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:46.797580 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:46.797545 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ebb197-56b8-4043-be23-682f002391b5-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:46.797748 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:46.797589 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9pc9\" (UniqueName: \"kubernetes.io/projected/a1ebb197-56b8-4043-be23-682f002391b5-kube-api-access-t9pc9\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:47.235014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:47.221468 5001 generic.go:356] "Generic (PLEG): container finished" podID="a1ebb197-56b8-4043-be23-682f002391b5" containerID="5ae1cf3f2009cca2ebdd16a3ff4d148b0fde0b6a4fdccf05a1a41870fda9b028" exitCode=0 Oct 30 00:38:47.235014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:47.221709 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-02665b-c768b5f9b-dpvxb" event={"ID":"a1ebb197-56b8-4043-be23-682f002391b5","Type":"ContainerDied","Data":"5ae1cf3f2009cca2ebdd16a3ff4d148b0fde0b6a4fdccf05a1a41870fda9b028"} Oct 30 00:38:47.235014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:47.221783 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-02665b-c768b5f9b-dpvxb" event={"ID":"a1ebb197-56b8-4043-be23-682f002391b5","Type":"ContainerDied","Data":"a14e77c6030501c2675538a41bc53e6641bbcc1f0e56ab617d506b8c6cffaafd"} Oct 30 00:38:47.235014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:47.221815 5001 scope.go:122] "RemoveContainer" containerID="5ae1cf3f2009cca2ebdd16a3ff4d148b0fde0b6a4fdccf05a1a41870fda9b028" Oct 30 00:38:47.235014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:47.222015 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-02665b-c768b5f9b-dpvxb" Oct 30 00:38:47.244601 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:47.244553 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-05c6b2-6f8558fd8b-kslr7" podUID="63d72c52-189d-4842-8946-d073e53b6879" containerName="payload-processing" containerID="cri-o://30937ffaa0502579eb4c311ae9511628d86d5ca3e656d84ff364d87e32d1443d" gracePeriod=25 Oct 30 00:38:47.248840 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:47.248818 5001 scope.go:122] "RemoveContainer" containerID="5ae1cf3f2009cca2ebdd16a3ff4d148b0fde0b6a4fdccf05a1a41870fda9b028" Oct 30 00:38:47.249599 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:38:47.249528 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ae1cf3f2009cca2ebdd16a3ff4d148b0fde0b6a4fdccf05a1a41870fda9b028\": container with ID starting with 5ae1cf3f2009cca2ebdd16a3ff4d148b0fde0b6a4fdccf05a1a41870fda9b028 not found: ID does not exist" containerID="5ae1cf3f2009cca2ebdd16a3ff4d148b0fde0b6a4fdccf05a1a41870fda9b028" Oct 30 00:38:47.249599 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:47.249571 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ae1cf3f2009cca2ebdd16a3ff4d148b0fde0b6a4fdccf05a1a41870fda9b028"} err="failed to get container status \"5ae1cf3f2009cca2ebdd16a3ff4d148b0fde0b6a4fdccf05a1a41870fda9b028\": rpc error: code = NotFound desc = could not find container \"5ae1cf3f2009cca2ebdd16a3ff4d148b0fde0b6a4fdccf05a1a41870fda9b028\": container with ID starting with 5ae1cf3f2009cca2ebdd16a3ff4d148b0fde0b6a4fdccf05a1a41870fda9b028 not found: ID does not exist" Oct 30 00:38:47.249980 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:47.249807 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-05c6b2-55b55d747-dgrp9" podUID="bd4c0988-ad1a-4288-85f9-573a02594fe5" containerName="payload-pre-processing" containerID="cri-o://0714c29b98a703ee902d129d3a2aae10482bf8d17f7a517136cc518206f53e8c" gracePeriod=25 Oct 30 00:38:47.274726 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:47.274695 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-02665b-c768b5f9b-dpvxb"] Oct 30 00:38:47.284346 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:47.284319 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-02665b-c768b5f9b-dpvxb"] Oct 30 00:38:47.324111 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:38:47.324079 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:38:48.327748 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:48.327709 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:38:48.328200 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:48.327765 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" podUID="e723059f-3ca0-45b1-a035-6ce96939fe4b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:38:48.919319 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:48.919274 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1ebb197-56b8-4043-be23-682f002391b5" path="/var/lib/kubelet/pods/a1ebb197-56b8-4043-be23-682f002391b5/volumes" Oct 30 00:38:49.124953 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:49.124906 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-415d24-krpp5"] Oct 30 00:38:49.135048 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:49.135014 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-415d24-krpp5"] Oct 30 00:38:49.195335 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:49.195248 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-415d24-8cc6f76cf-vnt5d" podUID="be687896-3ef5-4bfc-b170-3e91fcc13a3e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:38:49.196001 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:49.195976 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-05c6b2-55b55d747-dgrp9" podUID="bd4c0988-ad1a-4288-85f9-573a02594fe5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:38:50.238056 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:50.238017 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-415d24-9fd8c87c8-wn6vg" podUID="3f3b775e-8102-4e77-a2f7-272ac5d6ad5a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:38:50.919177 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:50.919141 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1" path="/var/lib/kubelet/pods/ce8cb122-4a11-4c0d-9c0f-7ebb7353b8c1/volumes" Oct 30 00:38:51.242213 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.242184 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:38:51.265433 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.265399 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-05c6b2-6f8558fd8b-kslr7" podUID="63d72c52-189d-4842-8946-d073e53b6879" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:38:51.281719 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.281689 5001 generic.go:356] "Generic (PLEG): container finished" podID="e723059f-3ca0-45b1-a035-6ce96939fe4b" containerID="3afb7dc28f6285c6b85072f7c8d2474f1a30607fc0a6fca1ec087c9231579375" exitCode=0 Oct 30 00:38:51.281899 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.281741 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" Oct 30 00:38:51.281899 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.281767 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" event={"ID":"e723059f-3ca0-45b1-a035-6ce96939fe4b","Type":"ContainerDied","Data":"3afb7dc28f6285c6b85072f7c8d2474f1a30607fc0a6fca1ec087c9231579375"} Oct 30 00:38:51.281899 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.281817 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk" event={"ID":"e723059f-3ca0-45b1-a035-6ce96939fe4b","Type":"ContainerDied","Data":"83e99d235bf8248957c960b446fb68bb9fbc6d6f28d2b740fc3b9982c258f70a"} Oct 30 00:38:51.281899 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.281839 5001 scope.go:122] "RemoveContainer" containerID="3afb7dc28f6285c6b85072f7c8d2474f1a30607fc0a6fca1ec087c9231579375" Oct 30 00:38:51.290210 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.290184 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-istio-envoy\") pod \"e723059f-3ca0-45b1-a035-6ce96939fe4b\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " Oct 30 00:38:51.290344 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.290279 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e723059f-3ca0-45b1-a035-6ce96939fe4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e723059f-3ca0-45b1-a035-6ce96939fe4b-istiod-ca-cert\") pod \"e723059f-3ca0-45b1-a035-6ce96939fe4b\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " Oct 30 00:38:51.290344 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.290329 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e723059f-3ca0-45b1-a035-6ce96939fe4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e723059f-3ca0-45b1-a035-6ce96939fe4b-istio-podinfo\") pod \"e723059f-3ca0-45b1-a035-6ce96939fe4b\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " Oct 30 00:38:51.290473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.290444 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-istio-data\") pod \"e723059f-3ca0-45b1-a035-6ce96939fe4b\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " Oct 30 00:38:51.290535 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.290484 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e723059f-3ca0-45b1-a035-6ce96939fe4b-kube-api-access-4wpqm\" (UniqueName: \"kubernetes.io/projected/e723059f-3ca0-45b1-a035-6ce96939fe4b-kube-api-access-4wpqm\") pod \"e723059f-3ca0-45b1-a035-6ce96939fe4b\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " Oct 30 00:38:51.290588 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.290547 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e723059f-3ca0-45b1-a035-6ce96939fe4b-istio-token\" (UniqueName: \"kubernetes.io/projected/e723059f-3ca0-45b1-a035-6ce96939fe4b-istio-token\") pod \"e723059f-3ca0-45b1-a035-6ce96939fe4b\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " Oct 30 00:38:51.290696 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.290624 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-workload-socket\") pod \"e723059f-3ca0-45b1-a035-6ce96939fe4b\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " Oct 30 00:38:51.290696 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.290672 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-credential-socket\") pod \"e723059f-3ca0-45b1-a035-6ce96939fe4b\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " Oct 30 00:38:51.290823 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.290693 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-istio-data" pod "e723059f-3ca0-45b1-a035-6ce96939fe4b" (UID: "e723059f-3ca0-45b1-a035-6ce96939fe4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:51.290823 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.290710 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e723059f-3ca0-45b1-a035-6ce96939fe4b-istiod-ca-cert" pod "e723059f-3ca0-45b1-a035-6ce96939fe4b" (UID: "e723059f-3ca0-45b1-a035-6ce96939fe4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:38:51.290823 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.290744 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-workload-certs\") pod \"e723059f-3ca0-45b1-a035-6ce96939fe4b\" (UID: \"e723059f-3ca0-45b1-a035-6ce96939fe4b\") " Oct 30 00:38:51.291011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.290970 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-workload-socket" pod "e723059f-3ca0-45b1-a035-6ce96939fe4b" (UID: "e723059f-3ca0-45b1-a035-6ce96939fe4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:51.291343 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.291318 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-credential-socket" pod "e723059f-3ca0-45b1-a035-6ce96939fe4b" (UID: "e723059f-3ca0-45b1-a035-6ce96939fe4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:51.291511 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.291419 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-workload-certs" pod "e723059f-3ca0-45b1-a035-6ce96939fe4b" (UID: "e723059f-3ca0-45b1-a035-6ce96939fe4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:51.291625 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.291404 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e723059f-3ca0-45b1-a035-6ce96939fe4b-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:51.291625 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.291626 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:51.291803 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.291643 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:51.292688 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.292662 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-istio-envoy" pod "e723059f-3ca0-45b1-a035-6ce96939fe4b" (UID: "e723059f-3ca0-45b1-a035-6ce96939fe4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:51.292778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.292702 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e723059f-3ca0-45b1-a035-6ce96939fe4b-istio-podinfo" pod "e723059f-3ca0-45b1-a035-6ce96939fe4b" (UID: "e723059f-3ca0-45b1-a035-6ce96939fe4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:38:51.293361 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.293339 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e723059f-3ca0-45b1-a035-6ce96939fe4b-kube-api-access-4wpqm" pod "e723059f-3ca0-45b1-a035-6ce96939fe4b" (UID: "e723059f-3ca0-45b1-a035-6ce96939fe4b"). InnerVolumeSpecName "kube-api-access-4wpqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:51.293408 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.293375 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e723059f-3ca0-45b1-a035-6ce96939fe4b-istio-token" pod "e723059f-3ca0-45b1-a035-6ce96939fe4b" (UID: "e723059f-3ca0-45b1-a035-6ce96939fe4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:51.294086 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.294070 5001 scope.go:122] "RemoveContainer" containerID="3afb7dc28f6285c6b85072f7c8d2474f1a30607fc0a6fca1ec087c9231579375" Oct 30 00:38:51.294370 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:38:51.294348 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3afb7dc28f6285c6b85072f7c8d2474f1a30607fc0a6fca1ec087c9231579375\": container with ID starting with 3afb7dc28f6285c6b85072f7c8d2474f1a30607fc0a6fca1ec087c9231579375 not found: ID does not exist" containerID="3afb7dc28f6285c6b85072f7c8d2474f1a30607fc0a6fca1ec087c9231579375" Oct 30 00:38:51.294424 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.294381 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3afb7dc28f6285c6b85072f7c8d2474f1a30607fc0a6fca1ec087c9231579375"} err="failed to get container status \"3afb7dc28f6285c6b85072f7c8d2474f1a30607fc0a6fca1ec087c9231579375\": rpc error: code = NotFound desc = could not find container \"3afb7dc28f6285c6b85072f7c8d2474f1a30607fc0a6fca1ec087c9231579375\": container with ID starting with 3afb7dc28f6285c6b85072f7c8d2474f1a30607fc0a6fca1ec087c9231579375 not found: ID does not exist" Oct 30 00:38:51.393028 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.392997 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e723059f-3ca0-45b1-a035-6ce96939fe4b-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:51.393028 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.393025 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wpqm\" (UniqueName: \"kubernetes.io/projected/e723059f-3ca0-45b1-a035-6ce96939fe4b-kube-api-access-4wpqm\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:51.393028 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.393036 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e723059f-3ca0-45b1-a035-6ce96939fe4b-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:51.393264 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.393045 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:51.393264 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.393059 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:51.393264 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.393067 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e723059f-3ca0-45b1-a035-6ce96939fe4b-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:38:51.631895 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.631850 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk"] Oct 30 00:38:51.642770 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:51.642735 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-02665b-openshift-default-748765b67-nf5tk"] Oct 30 00:38:52.878633 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:52.878589 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-05c6b2-kzxws"] Oct 30 00:38:52.887787 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:52.887760 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-05c6b2-kzxws"] Oct 30 00:38:52.919145 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:52.919111 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cf74bb9-4855-4415-ae14-301349aad6d3" path="/var/lib/kubelet/pods/5cf74bb9-4855-4415-ae14-301349aad6d3/volumes" Oct 30 00:38:52.919676 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:52.919661 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e723059f-3ca0-45b1-a035-6ce96939fe4b" path="/var/lib/kubelet/pods/e723059f-3ca0-45b1-a035-6ce96939fe4b/volumes" Oct 30 00:38:54.086951 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:54.086904 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-d96837-55b85d6d8c-tgmp9" Oct 30 00:38:54.087376 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:54.087000 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-d96837-5565b8fc78-qwjl5" Oct 30 00:38:56.973478 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:56.973449 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7505bd-v5nn2"] Oct 30 00:38:56.974051 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:56.974035 5001 cpu_manager.go:414] "containerMap: removing container" podUID="3a97385f-5ca1-41db-9a2a-66cbc70baad2" containerName="payload-pre-processing" Oct 30 00:38:56.974097 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:56.974051 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a97385f-5ca1-41db-9a2a-66cbc70baad2" containerName="payload-pre-processing" Oct 30 00:38:56.974097 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:56.974063 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea" containerName="payload-processing" Oct 30 00:38:56.974097 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:56.974068 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea" containerName="payload-processing" Oct 30 00:38:56.974097 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:56.974088 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e723059f-3ca0-45b1-a035-6ce96939fe4b" containerName="istio-proxy" Oct 30 00:38:56.974097 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:56.974093 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e723059f-3ca0-45b1-a035-6ce96939fe4b" containerName="istio-proxy" Oct 30 00:38:56.974268 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:56.974113 5001 cpu_manager.go:414] "containerMap: removing container" podUID="a1ebb197-56b8-4043-be23-682f002391b5" containerName="payload-processing" Oct 30 00:38:56.974268 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:56.974120 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1ebb197-56b8-4043-be23-682f002391b5" containerName="payload-processing" Oct 30 00:38:56.974268 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:56.974211 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a97385f-5ca1-41db-9a2a-66cbc70baad2" containerName="payload-pre-processing" Oct 30 00:38:56.974268 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:56.974217 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3b705e2-ecf3-4a35-a8e1-6111fd40a4ea" containerName="payload-processing" Oct 30 00:38:56.974268 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:56.974227 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e723059f-3ca0-45b1-a035-6ce96939fe4b" containerName="istio-proxy" Oct 30 00:38:56.974268 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:56.974237 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1ebb197-56b8-4043-be23-682f002391b5" containerName="payload-processing" Oct 30 00:38:56.979467 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:56.979446 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7505bd-v5nn2" Oct 30 00:38:56.993686 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:56.993649 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:38:57.003376 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:57.003341 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:38:57.003506 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:57.003399 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:38:57.003506 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:57.003449 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:38:57.008349 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:57.008321 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7505bd-v5nn2"] Oct 30 00:38:57.105067 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:57.105033 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv66d\" (UniqueName: \"kubernetes.io/projected/a0643035-da6e-4b23-93a6-04cccb0d9692-kube-api-access-nv66d\") pod \"maas-api-revoke-keys-e2e-worker-w0-7505bd-v5nn2\" (UID: \"a0643035-da6e-4b23-93a6-04cccb0d9692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7505bd-v5nn2" Oct 30 00:38:57.105259 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:57.105090 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0643035-da6e-4b23-93a6-04cccb0d9692-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7505bd-v5nn2\" (UID: \"a0643035-da6e-4b23-93a6-04cccb0d9692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7505bd-v5nn2" Oct 30 00:38:57.105259 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:57.105239 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0643035-da6e-4b23-93a6-04cccb0d9692-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7505bd-v5nn2\" (UID: \"a0643035-da6e-4b23-93a6-04cccb0d9692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7505bd-v5nn2" Oct 30 00:38:57.206467 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:57.206431 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0643035-da6e-4b23-93a6-04cccb0d9692-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7505bd-v5nn2\" (UID: \"a0643035-da6e-4b23-93a6-04cccb0d9692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7505bd-v5nn2" Oct 30 00:38:57.206696 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:57.206586 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv66d\" (UniqueName: \"kubernetes.io/projected/a0643035-da6e-4b23-93a6-04cccb0d9692-kube-api-access-nv66d\") pod \"maas-api-revoke-keys-e2e-worker-w0-7505bd-v5nn2\" (UID: \"a0643035-da6e-4b23-93a6-04cccb0d9692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7505bd-v5nn2" Oct 30 00:38:57.206696 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:57.206634 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0643035-da6e-4b23-93a6-04cccb0d9692-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7505bd-v5nn2\" (UID: \"a0643035-da6e-4b23-93a6-04cccb0d9692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7505bd-v5nn2" Oct 30 00:38:57.207127 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:57.207099 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0643035-da6e-4b23-93a6-04cccb0d9692-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7505bd-v5nn2\" (UID: \"a0643035-da6e-4b23-93a6-04cccb0d9692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7505bd-v5nn2" Oct 30 00:38:57.209178 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:57.209151 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0643035-da6e-4b23-93a6-04cccb0d9692-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7505bd-v5nn2\" (UID: \"a0643035-da6e-4b23-93a6-04cccb0d9692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7505bd-v5nn2" Oct 30 00:38:57.227479 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:57.227420 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv66d\" (UniqueName: \"kubernetes.io/projected/a0643035-da6e-4b23-93a6-04cccb0d9692-kube-api-access-nv66d\") pod \"maas-api-revoke-keys-e2e-worker-w0-7505bd-v5nn2\" (UID: \"a0643035-da6e-4b23-93a6-04cccb0d9692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7505bd-v5nn2" Oct 30 00:38:57.294305 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:57.294271 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7505bd-v5nn2" Oct 30 00:38:57.604785 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:38:57.604754 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0643035_da6e_4b23_93a6_04cccb0d9692.slice/crio-4e259fe30485c4a1869c97004a9826c7efcbf204fe5a3a3a067b0d0dab213351 WatchSource:0}: Error finding container 4e259fe30485c4a1869c97004a9826c7efcbf204fe5a3a3a067b0d0dab213351: Status 404 returned error can't find the container with id 4e259fe30485c4a1869c97004a9826c7efcbf204fe5a3a3a067b0d0dab213351 Oct 30 00:38:57.605564 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:57.605541 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7505bd-v5nn2"] Oct 30 00:38:58.387311 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:58.387267 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7505bd-v5nn2" event={"ID":"a0643035-da6e-4b23-93a6-04cccb0d9692","Type":"ContainerStarted","Data":"d44c21fe5c583641f9bf5ddb1f0fdd098dba1a06a4c63838e8b81e721f06b4cf"} Oct 30 00:38:58.387768 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:58.387336 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7505bd-v5nn2" event={"ID":"a0643035-da6e-4b23-93a6-04cccb0d9692","Type":"ContainerStarted","Data":"4e259fe30485c4a1869c97004a9826c7efcbf204fe5a3a3a067b0d0dab213351"} Oct 30 00:38:58.422717 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:58.422661 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7505bd-v5nn2" podStartSLOduration=2.422645175 podStartE2EDuration="2.422645175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:58.422183163 +0000 UTC m=+2053.627684345" watchObservedRunningTime="2026-10-30 00:38:58.422645175 +0000 UTC m=+2053.628146355" Oct 30 00:38:59.195468 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:59.195427 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-415d24-8cc6f76cf-vnt5d" podUID="be687896-3ef5-4bfc-b170-3e91fcc13a3e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:38:59.195890 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:59.195862 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-05c6b2-55b55d747-dgrp9" podUID="bd4c0988-ad1a-4288-85f9-573a02594fe5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:38:59.397320 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:59.397287 5001 generic.go:356] "Generic (PLEG): container finished" podID="a0643035-da6e-4b23-93a6-04cccb0d9692" containerID="d44c21fe5c583641f9bf5ddb1f0fdd098dba1a06a4c63838e8b81e721f06b4cf" exitCode=0 Oct 30 00:38:59.397793 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:38:59.397382 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7505bd-v5nn2" event={"ID":"a0643035-da6e-4b23-93a6-04cccb0d9692","Type":"ContainerDied","Data":"d44c21fe5c583641f9bf5ddb1f0fdd098dba1a06a4c63838e8b81e721f06b4cf"} Oct 30 00:39:00.238414 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:00.238376 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-415d24-9fd8c87c8-wn6vg" podUID="3f3b775e-8102-4e77-a2f7-272ac5d6ad5a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:39:00.548960 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:00.548900 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7505bd-v5nn2" Oct 30 00:39:00.607045 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:00.607001 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-322d20-r8f66"] Oct 30 00:39:00.607969 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:00.607948 5001 cpu_manager.go:414] "containerMap: removing container" podUID="a0643035-da6e-4b23-93a6-04cccb0d9692" containerName="revoke-keys" Oct 30 00:39:00.608031 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:00.607972 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0643035-da6e-4b23-93a6-04cccb0d9692" containerName="revoke-keys" Oct 30 00:39:00.608238 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:00.608218 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0643035-da6e-4b23-93a6-04cccb0d9692" containerName="revoke-keys" Oct 30 00:39:00.612463 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:00.612439 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-322d20-r8f66" Oct 30 00:39:00.632714 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:00.632685 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-322d20-r8f66"] Oct 30 00:39:00.654051 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:00.654026 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0643035-da6e-4b23-93a6-04cccb0d9692-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0643035-da6e-4b23-93a6-04cccb0d9692-maas-api-xks-ca\") pod \"a0643035-da6e-4b23-93a6-04cccb0d9692\" (UID: \"a0643035-da6e-4b23-93a6-04cccb0d9692\") " Oct 30 00:39:00.654197 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:00.654075 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0643035-da6e-4b23-93a6-04cccb0d9692-kube-api-access-nv66d\" (UniqueName: \"kubernetes.io/projected/a0643035-da6e-4b23-93a6-04cccb0d9692-kube-api-access-nv66d\") pod \"a0643035-da6e-4b23-93a6-04cccb0d9692\" (UID: \"a0643035-da6e-4b23-93a6-04cccb0d9692\") " Oct 30 00:39:00.654364 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:00.654205 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0643035-da6e-4b23-93a6-04cccb0d9692-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0643035-da6e-4b23-93a6-04cccb0d9692-maas-api-openshift-ca\") pod \"a0643035-da6e-4b23-93a6-04cccb0d9692\" (UID: \"a0643035-da6e-4b23-93a6-04cccb0d9692\") " Oct 30 00:39:00.654842 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:00.654815 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0643035-da6e-4b23-93a6-04cccb0d9692-maas-api-openshift-ca" pod "a0643035-da6e-4b23-93a6-04cccb0d9692" (UID: "a0643035-da6e-4b23-93a6-04cccb0d9692"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:00.655286 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:00.655258 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0643035-da6e-4b23-93a6-04cccb0d9692-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:00.657404 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:00.657370 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0643035-da6e-4b23-93a6-04cccb0d9692-maas-api-xks-ca" pod "a0643035-da6e-4b23-93a6-04cccb0d9692" (UID: "a0643035-da6e-4b23-93a6-04cccb0d9692"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:39:00.658068 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:00.658043 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0643035-da6e-4b23-93a6-04cccb0d9692-kube-api-access-nv66d" pod "a0643035-da6e-4b23-93a6-04cccb0d9692" (UID: "a0643035-da6e-4b23-93a6-04cccb0d9692"). InnerVolumeSpecName "kube-api-access-nv66d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:00.756159 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:00.756123 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqr62\" (UniqueName: \"kubernetes.io/projected/368f7283-c28c-4ae5-859c-af2488a5c486-kube-api-access-jqr62\") pod \"maas-api-revoke-keys-e2e-worker-w2-322d20-r8f66\" (UID: \"368f7283-c28c-4ae5-859c-af2488a5c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-322d20-r8f66" Oct 30 00:39:00.756328 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:00.756242 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/368f7283-c28c-4ae5-859c-af2488a5c486-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-322d20-r8f66\" (UID: \"368f7283-c28c-4ae5-859c-af2488a5c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-322d20-r8f66" Oct 30 00:39:00.756384 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:00.756361 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/368f7283-c28c-4ae5-859c-af2488a5c486-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-322d20-r8f66\" (UID: \"368f7283-c28c-4ae5-859c-af2488a5c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-322d20-r8f66" Oct 30 00:39:00.756536 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:00.756518 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0643035-da6e-4b23-93a6-04cccb0d9692-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:00.756574 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:00.756540 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv66d\" (UniqueName: \"kubernetes.io/projected/a0643035-da6e-4b23-93a6-04cccb0d9692-kube-api-access-nv66d\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:00.858182 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:00.858098 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/368f7283-c28c-4ae5-859c-af2488a5c486-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-322d20-r8f66\" (UID: \"368f7283-c28c-4ae5-859c-af2488a5c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-322d20-r8f66" Oct 30 00:39:00.858367 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:00.858226 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqr62\" (UniqueName: \"kubernetes.io/projected/368f7283-c28c-4ae5-859c-af2488a5c486-kube-api-access-jqr62\") pod \"maas-api-revoke-keys-e2e-worker-w2-322d20-r8f66\" (UID: \"368f7283-c28c-4ae5-859c-af2488a5c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-322d20-r8f66" Oct 30 00:39:00.858367 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:00.858334 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/368f7283-c28c-4ae5-859c-af2488a5c486-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-322d20-r8f66\" (UID: \"368f7283-c28c-4ae5-859c-af2488a5c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-322d20-r8f66" Oct 30 00:39:00.858991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:00.858958 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/368f7283-c28c-4ae5-859c-af2488a5c486-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-322d20-r8f66\" (UID: \"368f7283-c28c-4ae5-859c-af2488a5c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-322d20-r8f66" Oct 30 00:39:00.860615 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:00.860585 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/368f7283-c28c-4ae5-859c-af2488a5c486-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-322d20-r8f66\" (UID: \"368f7283-c28c-4ae5-859c-af2488a5c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-322d20-r8f66" Oct 30 00:39:00.878906 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:00.878882 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqr62\" (UniqueName: \"kubernetes.io/projected/368f7283-c28c-4ae5-859c-af2488a5c486-kube-api-access-jqr62\") pod \"maas-api-revoke-keys-e2e-worker-w2-322d20-r8f66\" (UID: \"368f7283-c28c-4ae5-859c-af2488a5c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-322d20-r8f66" Oct 30 00:39:00.927821 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:00.927798 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-322d20-r8f66" Oct 30 00:39:01.233384 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:39:01.233356 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod368f7283_c28c_4ae5_859c_af2488a5c486.slice/crio-0aef78da9b5b13377c3834ca6d17db4ae5fcc072e698467610bfb55a1ad354c3 WatchSource:0}: Error finding container 0aef78da9b5b13377c3834ca6d17db4ae5fcc072e698467610bfb55a1ad354c3: Status 404 returned error can't find the container with id 0aef78da9b5b13377c3834ca6d17db4ae5fcc072e698467610bfb55a1ad354c3 Oct 30 00:39:01.237017 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:01.235832 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-322d20-r8f66"] Oct 30 00:39:01.265084 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:01.265047 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-05c6b2-6f8558fd8b-kslr7" podUID="63d72c52-189d-4842-8946-d073e53b6879" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:39:01.405448 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:01.405417 5001 scope.go:122] "RemoveContainer" containerID="bcbc2a74bd72da2d3dc5c806eccd3d4e96061472f7bc0a0418c5c7792457dfb6" Oct 30 00:39:01.417978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:01.417823 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-322d20-r8f66" event={"ID":"368f7283-c28c-4ae5-859c-af2488a5c486","Type":"ContainerStarted","Data":"0aef78da9b5b13377c3834ca6d17db4ae5fcc072e698467610bfb55a1ad354c3"} Oct 30 00:39:01.417978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:01.417959 5001 scope.go:122] "RemoveContainer" containerID="14fa5ee31c34fa120e28c8f1b1029476ec09ade105099cac16c2a6192be1bc6b" Oct 30 00:39:01.419506 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:01.419485 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7505bd-v5nn2" Oct 30 00:39:01.419619 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:01.419513 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7505bd-v5nn2" event={"ID":"a0643035-da6e-4b23-93a6-04cccb0d9692","Type":"ContainerDied","Data":"4e259fe30485c4a1869c97004a9826c7efcbf204fe5a3a3a067b0d0dab213351"} Oct 30 00:39:01.419619 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:01.419551 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e259fe30485c4a1869c97004a9826c7efcbf204fe5a3a3a067b0d0dab213351" Oct 30 00:39:01.435097 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:01.435076 5001 scope.go:122] "RemoveContainer" containerID="cbff866fef072bd37e2287c0ceca83485cf5324e182f38fcd3fa97910f69b0d8" Oct 30 00:39:01.445655 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:01.445634 5001 scope.go:122] "RemoveContainer" containerID="2cdaf15df360e278022cb90a80d05c2b8031f6f658894c01c5b65b66839c5732" Oct 30 00:39:01.455585 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:01.455556 5001 scope.go:122] "RemoveContainer" containerID="54fc1e319135f739cd00341b547b519f8ba61ba86ce7a45dc3d9f5c9fe2fef73" Oct 30 00:39:01.465293 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:01.465277 5001 scope.go:122] "RemoveContainer" containerID="67dce500ec036f2aab42964ee8aae4f3307c6b18a03550b56254a6a84256e2cb" Oct 30 00:39:01.474703 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:01.474683 5001 scope.go:122] "RemoveContainer" containerID="2b12a2d48fde9fbd26a91142b1c666984f10c21009c12fc7d5282f35b534fe29" Oct 30 00:39:01.483992 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:01.483971 5001 scope.go:122] "RemoveContainer" containerID="647a01a83022f6a61d2e8410720a2864332b4536d0ca6ba210b67eef678e81b7" Oct 30 00:39:01.493291 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:01.493275 5001 scope.go:122] "RemoveContainer" containerID="efc0d87efc297068e8a20f038be19301da0e59498f27199413b2b67ec19be094" Oct 30 00:39:01.502485 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:01.502462 5001 scope.go:122] "RemoveContainer" containerID="dad740d78220851e665c0f5a2febdc8c994ab915045a57478d0ad774a84e9cda" Oct 30 00:39:01.754771 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:01.754691 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv"] Oct 30 00:39:02.430941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:02.430895 5001 generic.go:356] "Generic (PLEG): container finished" podID="368f7283-c28c-4ae5-859c-af2488a5c486" containerID="f9c870c350bab09e7a032ddfb9c2932d98f23905b08f7ac48381b4ff840817a6" exitCode=0 Oct 30 00:39:02.431117 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:02.430961 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-322d20-r8f66" event={"ID":"368f7283-c28c-4ae5-859c-af2488a5c486","Type":"ContainerDied","Data":"f9c870c350bab09e7a032ddfb9c2932d98f23905b08f7ac48381b4ff840817a6"} Oct 30 00:39:02.655395 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:02.655370 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-415d24-9fd8c87c8-wn6vg" Oct 30 00:39:02.699428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:02.699403 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-415d24-8cc6f76cf-vnt5d" Oct 30 00:39:02.781785 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:02.781752 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f3b775e-8102-4e77-a2f7-272ac5d6ad5a-kube-api-access-9hnkf\" (UniqueName: \"kubernetes.io/projected/3f3b775e-8102-4e77-a2f7-272ac5d6ad5a-kube-api-access-9hnkf\") pod \"3f3b775e-8102-4e77-a2f7-272ac5d6ad5a\" (UID: \"3f3b775e-8102-4e77-a2f7-272ac5d6ad5a\") " Oct 30 00:39:02.782272 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:02.781815 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f3b775e-8102-4e77-a2f7-272ac5d6ad5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f3b775e-8102-4e77-a2f7-272ac5d6ad5a-plugins-config-volume\") pod \"3f3b775e-8102-4e77-a2f7-272ac5d6ad5a\" (UID: \"3f3b775e-8102-4e77-a2f7-272ac5d6ad5a\") " Oct 30 00:39:02.782272 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:02.782000 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be687896-3ef5-4bfc-b170-3e91fcc13a3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be687896-3ef5-4bfc-b170-3e91fcc13a3e-plugins-config-volume\") pod \"be687896-3ef5-4bfc-b170-3e91fcc13a3e\" (UID: \"be687896-3ef5-4bfc-b170-3e91fcc13a3e\") " Oct 30 00:39:02.782272 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:02.782141 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be687896-3ef5-4bfc-b170-3e91fcc13a3e-kube-api-access-vnpss\" (UniqueName: \"kubernetes.io/projected/be687896-3ef5-4bfc-b170-3e91fcc13a3e-kube-api-access-vnpss\") pod \"be687896-3ef5-4bfc-b170-3e91fcc13a3e\" (UID: \"be687896-3ef5-4bfc-b170-3e91fcc13a3e\") " Oct 30 00:39:02.782464 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:02.782308 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f3b775e-8102-4e77-a2f7-272ac5d6ad5a-plugins-config-volume" pod "3f3b775e-8102-4e77-a2f7-272ac5d6ad5a" (UID: "3f3b775e-8102-4e77-a2f7-272ac5d6ad5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:02.782464 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:02.782398 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be687896-3ef5-4bfc-b170-3e91fcc13a3e-plugins-config-volume" pod "be687896-3ef5-4bfc-b170-3e91fcc13a3e" (UID: "be687896-3ef5-4bfc-b170-3e91fcc13a3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:02.782710 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:02.782687 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f3b775e-8102-4e77-a2f7-272ac5d6ad5a-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:02.782803 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:02.782715 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be687896-3ef5-4bfc-b170-3e91fcc13a3e-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:02.784337 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:02.784312 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be687896-3ef5-4bfc-b170-3e91fcc13a3e-kube-api-access-vnpss" pod "be687896-3ef5-4bfc-b170-3e91fcc13a3e" (UID: "be687896-3ef5-4bfc-b170-3e91fcc13a3e"). InnerVolumeSpecName "kube-api-access-vnpss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:02.784801 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:02.784773 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f3b775e-8102-4e77-a2f7-272ac5d6ad5a-kube-api-access-9hnkf" pod "3f3b775e-8102-4e77-a2f7-272ac5d6ad5a" (UID: "3f3b775e-8102-4e77-a2f7-272ac5d6ad5a"). InnerVolumeSpecName "kube-api-access-9hnkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:02.835261 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:02.835231 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf"] Oct 30 00:39:02.835496 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:02.835465 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" podUID="1222e77c-6ae5-4ce1-a15f-caa491260730" containerName="istio-proxy" containerID="cri-o://6cc703f7001268a7b52c39bea2e3c7a0fbd6c65f5455116e64b37e03d1346ee3" gracePeriod=30 Oct 30 00:39:02.840428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:02.840401 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m"] Oct 30 00:39:02.840632 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:02.840609 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" podUID="44355452-926f-4608-b0eb-24bb7e30dcbd" containerName="istio-proxy" containerID="cri-o://379b761ea39da3c3064a149b8cc39298798c811cf0ad42d8d67857184eedade5" gracePeriod=30 Oct 30 00:39:02.884004 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:02.883889 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnpss\" (UniqueName: \"kubernetes.io/projected/be687896-3ef5-4bfc-b170-3e91fcc13a3e-kube-api-access-vnpss\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:02.884004 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:02.883947 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hnkf\" (UniqueName: \"kubernetes.io/projected/3f3b775e-8102-4e77-a2f7-272ac5d6ad5a-kube-api-access-9hnkf\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:03.441229 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:03.441187 5001 generic.go:356] "Generic (PLEG): container finished" podID="3f3b775e-8102-4e77-a2f7-272ac5d6ad5a" containerID="5910f3d6ff3668603ced6ac7197ee3fad977de116c28dbcc9f2fc1a6d5f13dce" exitCode=0 Oct 30 00:39:03.441407 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:03.441233 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-415d24-9fd8c87c8-wn6vg" Oct 30 00:39:03.441407 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:03.441229 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-415d24-9fd8c87c8-wn6vg" event={"ID":"3f3b775e-8102-4e77-a2f7-272ac5d6ad5a","Type":"ContainerDied","Data":"5910f3d6ff3668603ced6ac7197ee3fad977de116c28dbcc9f2fc1a6d5f13dce"} Oct 30 00:39:03.441407 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:03.441351 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-415d24-9fd8c87c8-wn6vg" event={"ID":"3f3b775e-8102-4e77-a2f7-272ac5d6ad5a","Type":"ContainerDied","Data":"074b5448875a1fbd36d5ac7e128da5c6fe5a80f737d583be7bcb8d5cb4d50fae"} Oct 30 00:39:03.441407 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:03.441375 5001 scope.go:122] "RemoveContainer" containerID="5910f3d6ff3668603ced6ac7197ee3fad977de116c28dbcc9f2fc1a6d5f13dce" Oct 30 00:39:03.443007 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:03.442888 5001 generic.go:356] "Generic (PLEG): container finished" podID="be687896-3ef5-4bfc-b170-3e91fcc13a3e" containerID="84298f705f2b88ee0b66b590765433bfb83e3b2f4996c5e0600fa5a98f2ff830" exitCode=0 Oct 30 00:39:03.443007 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:03.442962 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-415d24-8cc6f76cf-vnt5d" Oct 30 00:39:03.443007 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:03.442977 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-415d24-8cc6f76cf-vnt5d" event={"ID":"be687896-3ef5-4bfc-b170-3e91fcc13a3e","Type":"ContainerDied","Data":"84298f705f2b88ee0b66b590765433bfb83e3b2f4996c5e0600fa5a98f2ff830"} Oct 30 00:39:03.443237 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:03.443029 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-415d24-8cc6f76cf-vnt5d" event={"ID":"be687896-3ef5-4bfc-b170-3e91fcc13a3e","Type":"ContainerDied","Data":"01a48f11bc6804fc88fa9ac4a9f7b710e63de556d1ad2bc26d3b484c98202b7e"} Oct 30 00:39:03.453347 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:03.453322 5001 scope.go:122] "RemoveContainer" containerID="5910f3d6ff3668603ced6ac7197ee3fad977de116c28dbcc9f2fc1a6d5f13dce" Oct 30 00:39:03.453682 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:39:03.453647 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5910f3d6ff3668603ced6ac7197ee3fad977de116c28dbcc9f2fc1a6d5f13dce\": container with ID starting with 5910f3d6ff3668603ced6ac7197ee3fad977de116c28dbcc9f2fc1a6d5f13dce not found: ID does not exist" containerID="5910f3d6ff3668603ced6ac7197ee3fad977de116c28dbcc9f2fc1a6d5f13dce" Oct 30 00:39:03.453739 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:03.453697 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5910f3d6ff3668603ced6ac7197ee3fad977de116c28dbcc9f2fc1a6d5f13dce"} err="failed to get container status \"5910f3d6ff3668603ced6ac7197ee3fad977de116c28dbcc9f2fc1a6d5f13dce\": rpc error: code = NotFound desc = could not find container \"5910f3d6ff3668603ced6ac7197ee3fad977de116c28dbcc9f2fc1a6d5f13dce\": container with ID starting with 5910f3d6ff3668603ced6ac7197ee3fad977de116c28dbcc9f2fc1a6d5f13dce not found: ID does not exist" Oct 30 00:39:03.453739 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:03.453720 5001 scope.go:122] "RemoveContainer" containerID="84298f705f2b88ee0b66b590765433bfb83e3b2f4996c5e0600fa5a98f2ff830" Oct 30 00:39:03.464724 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:03.464704 5001 scope.go:122] "RemoveContainer" containerID="84298f705f2b88ee0b66b590765433bfb83e3b2f4996c5e0600fa5a98f2ff830" Oct 30 00:39:03.465057 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:39:03.465033 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84298f705f2b88ee0b66b590765433bfb83e3b2f4996c5e0600fa5a98f2ff830\": container with ID starting with 84298f705f2b88ee0b66b590765433bfb83e3b2f4996c5e0600fa5a98f2ff830 not found: ID does not exist" containerID="84298f705f2b88ee0b66b590765433bfb83e3b2f4996c5e0600fa5a98f2ff830" Oct 30 00:39:03.465141 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:03.465068 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84298f705f2b88ee0b66b590765433bfb83e3b2f4996c5e0600fa5a98f2ff830"} err="failed to get container status \"84298f705f2b88ee0b66b590765433bfb83e3b2f4996c5e0600fa5a98f2ff830\": rpc error: code = NotFound desc = could not find container \"84298f705f2b88ee0b66b590765433bfb83e3b2f4996c5e0600fa5a98f2ff830\": container with ID starting with 84298f705f2b88ee0b66b590765433bfb83e3b2f4996c5e0600fa5a98f2ff830 not found: ID does not exist" Oct 30 00:39:03.489500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:03.489467 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-415d24-9fd8c87c8-wn6vg"] Oct 30 00:39:03.493651 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:03.493621 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-415d24-9fd8c87c8-wn6vg"] Oct 30 00:39:03.520003 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:03.519974 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-415d24-8cc6f76cf-vnt5d"] Oct 30 00:39:03.524360 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:03.524335 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-415d24-8cc6f76cf-vnt5d"] Oct 30 00:39:03.595503 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:03.595476 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-322d20-r8f66" Oct 30 00:39:03.693707 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:03.693626 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/368f7283-c28c-4ae5-859c-af2488a5c486-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/368f7283-c28c-4ae5-859c-af2488a5c486-maas-api-openshift-ca\") pod \"368f7283-c28c-4ae5-859c-af2488a5c486\" (UID: \"368f7283-c28c-4ae5-859c-af2488a5c486\") " Oct 30 00:39:03.693866 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:03.693781 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/368f7283-c28c-4ae5-859c-af2488a5c486-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/368f7283-c28c-4ae5-859c-af2488a5c486-maas-api-xks-ca\") pod \"368f7283-c28c-4ae5-859c-af2488a5c486\" (UID: \"368f7283-c28c-4ae5-859c-af2488a5c486\") " Oct 30 00:39:03.693866 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:03.693831 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/368f7283-c28c-4ae5-859c-af2488a5c486-kube-api-access-jqr62\" (UniqueName: \"kubernetes.io/projected/368f7283-c28c-4ae5-859c-af2488a5c486-kube-api-access-jqr62\") pod \"368f7283-c28c-4ae5-859c-af2488a5c486\" (UID: \"368f7283-c28c-4ae5-859c-af2488a5c486\") " Oct 30 00:39:03.694038 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:03.694013 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/368f7283-c28c-4ae5-859c-af2488a5c486-maas-api-openshift-ca" pod "368f7283-c28c-4ae5-859c-af2488a5c486" (UID: "368f7283-c28c-4ae5-859c-af2488a5c486"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:03.694336 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:03.694317 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/368f7283-c28c-4ae5-859c-af2488a5c486-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:03.695926 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:03.695891 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/368f7283-c28c-4ae5-859c-af2488a5c486-maas-api-xks-ca" pod "368f7283-c28c-4ae5-859c-af2488a5c486" (UID: "368f7283-c28c-4ae5-859c-af2488a5c486"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:39:03.696056 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:03.695900 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/368f7283-c28c-4ae5-859c-af2488a5c486-kube-api-access-jqr62" pod "368f7283-c28c-4ae5-859c-af2488a5c486" (UID: "368f7283-c28c-4ae5-859c-af2488a5c486"). InnerVolumeSpecName "kube-api-access-jqr62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:03.796086 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:03.796037 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/368f7283-c28c-4ae5-859c-af2488a5c486-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:03.796086 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:03.796080 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqr62\" (UniqueName: \"kubernetes.io/projected/368f7283-c28c-4ae5-859c-af2488a5c486-kube-api-access-jqr62\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:04.452074 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:04.452046 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-322d20-r8f66" Oct 30 00:39:04.452267 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:04.452084 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-322d20-r8f66" event={"ID":"368f7283-c28c-4ae5-859c-af2488a5c486","Type":"ContainerDied","Data":"0aef78da9b5b13377c3834ca6d17db4ae5fcc072e698467610bfb55a1ad354c3"} Oct 30 00:39:04.452267 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:04.452135 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0aef78da9b5b13377c3834ca6d17db4ae5fcc072e698467610bfb55a1ad354c3" Oct 30 00:39:04.939480 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:04.939448 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f3b775e-8102-4e77-a2f7-272ac5d6ad5a" path="/var/lib/kubelet/pods/3f3b775e-8102-4e77-a2f7-272ac5d6ad5a/volumes" Oct 30 00:39:04.939974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:04.939957 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be687896-3ef5-4bfc-b170-3e91fcc13a3e" path="/var/lib/kubelet/pods/be687896-3ef5-4bfc-b170-3e91fcc13a3e/volumes" Oct 30 00:39:05.027672 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:05.027636 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cee150a0-qkd8t"] Oct 30 00:39:05.031848 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:05.031816 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cee150a0-qkd8t"] Oct 30 00:39:06.919042 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:06.919008 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a4eee95-140b-45a8-a9a8-0c5b2be11e65" path="/var/lib/kubelet/pods/2a4eee95-140b-45a8-a9a8-0c5b2be11e65/volumes" Oct 30 00:39:07.563212 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:07.563101 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7505bd-65fc9799fb-8p95v"] Oct 30 00:39:07.569814 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:07.566806 5001 generic.go:356] "Generic (PLEG): container finished" podID="bd4c0988-ad1a-4288-85f9-573a02594fe5" containerID="0714c29b98a703ee902d129d3a2aae10482bf8d17f7a517136cc518206f53e8c" exitCode=0 Oct 30 00:39:07.569814 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:07.566885 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-05c6b2-55b55d747-dgrp9" event={"ID":"bd4c0988-ad1a-4288-85f9-573a02594fe5","Type":"ContainerDied","Data":"0714c29b98a703ee902d129d3a2aae10482bf8d17f7a517136cc518206f53e8c"} Oct 30 00:39:07.569814 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:07.568490 5001 generic.go:356] "Generic (PLEG): container finished" podID="63d72c52-189d-4842-8946-d073e53b6879" containerID="30937ffaa0502579eb4c311ae9511628d86d5ca3e656d84ff364d87e32d1443d" exitCode=0 Oct 30 00:39:07.569814 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:07.568631 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-05c6b2-6f8558fd8b-kslr7" event={"ID":"63d72c52-189d-4842-8946-d073e53b6879","Type":"ContainerDied","Data":"30937ffaa0502579eb4c311ae9511628d86d5ca3e656d84ff364d87e32d1443d"} Oct 30 00:39:07.571266 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:07.570006 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7505bd-76fc47c6d7-jqkf2"] Oct 30 00:39:07.661902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:07.661875 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-05c6b2-55b55d747-dgrp9" Oct 30 00:39:07.732191 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:07.732147 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-05c6b2-6f8558fd8b-kslr7" Oct 30 00:39:07.760190 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:07.760157 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd4c0988-ad1a-4288-85f9-573a02594fe5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd4c0988-ad1a-4288-85f9-573a02594fe5-plugins-config-volume\") pod \"bd4c0988-ad1a-4288-85f9-573a02594fe5\" (UID: \"bd4c0988-ad1a-4288-85f9-573a02594fe5\") " Oct 30 00:39:07.760347 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:07.760278 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63d72c52-189d-4842-8946-d073e53b6879-kube-api-access-424mg\" (UniqueName: \"kubernetes.io/projected/63d72c52-189d-4842-8946-d073e53b6879-kube-api-access-424mg\") pod \"63d72c52-189d-4842-8946-d073e53b6879\" (UID: \"63d72c52-189d-4842-8946-d073e53b6879\") " Oct 30 00:39:07.760402 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:07.760352 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63d72c52-189d-4842-8946-d073e53b6879-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63d72c52-189d-4842-8946-d073e53b6879-plugins-config-volume\") pod \"63d72c52-189d-4842-8946-d073e53b6879\" (UID: \"63d72c52-189d-4842-8946-d073e53b6879\") " Oct 30 00:39:07.760484 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:07.760464 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd4c0988-ad1a-4288-85f9-573a02594fe5-kube-api-access-wwjqw\" (UniqueName: \"kubernetes.io/projected/bd4c0988-ad1a-4288-85f9-573a02594fe5-kube-api-access-wwjqw\") pod \"bd4c0988-ad1a-4288-85f9-573a02594fe5\" (UID: \"bd4c0988-ad1a-4288-85f9-573a02594fe5\") " Oct 30 00:39:07.760734 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:07.760693 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd4c0988-ad1a-4288-85f9-573a02594fe5-plugins-config-volume" pod "bd4c0988-ad1a-4288-85f9-573a02594fe5" (UID: "bd4c0988-ad1a-4288-85f9-573a02594fe5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:07.760861 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:07.760771 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63d72c52-189d-4842-8946-d073e53b6879-plugins-config-volume" pod "63d72c52-189d-4842-8946-d073e53b6879" (UID: "63d72c52-189d-4842-8946-d073e53b6879"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:07.761023 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:07.760994 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd4c0988-ad1a-4288-85f9-573a02594fe5-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:07.761083 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:07.761022 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63d72c52-189d-4842-8946-d073e53b6879-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:07.762607 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:07.762578 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd4c0988-ad1a-4288-85f9-573a02594fe5-kube-api-access-wwjqw" pod "bd4c0988-ad1a-4288-85f9-573a02594fe5" (UID: "bd4c0988-ad1a-4288-85f9-573a02594fe5"). InnerVolumeSpecName "kube-api-access-wwjqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:07.763026 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:07.763001 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d72c52-189d-4842-8946-d073e53b6879-kube-api-access-424mg" pod "63d72c52-189d-4842-8946-d073e53b6879" (UID: "63d72c52-189d-4842-8946-d073e53b6879"). InnerVolumeSpecName "kube-api-access-424mg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:07.862626 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:07.862598 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-424mg\" (UniqueName: \"kubernetes.io/projected/63d72c52-189d-4842-8946-d073e53b6879-kube-api-access-424mg\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:07.862626 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:07.862627 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwjqw\" (UniqueName: \"kubernetes.io/projected/bd4c0988-ad1a-4288-85f9-573a02594fe5-kube-api-access-wwjqw\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:08.089054 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.089027 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:39:08.187981 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.185350 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1222e77c-6ae5-4ce1-a15f-caa491260730-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1222e77c-6ae5-4ce1-a15f-caa491260730-istio-podinfo\") pod \"1222e77c-6ae5-4ce1-a15f-caa491260730\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " Oct 30 00:39:08.187981 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.185450 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-workload-socket\") pod \"1222e77c-6ae5-4ce1-a15f-caa491260730\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " Oct 30 00:39:08.187981 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.185480 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1222e77c-6ae5-4ce1-a15f-caa491260730-istio-token\" (UniqueName: \"kubernetes.io/projected/1222e77c-6ae5-4ce1-a15f-caa491260730-istio-token\") pod \"1222e77c-6ae5-4ce1-a15f-caa491260730\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " Oct 30 00:39:08.187981 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.185536 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1222e77c-6ae5-4ce1-a15f-caa491260730-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1222e77c-6ae5-4ce1-a15f-caa491260730-istiod-ca-cert\") pod \"1222e77c-6ae5-4ce1-a15f-caa491260730\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " Oct 30 00:39:08.187981 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.185620 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-istio-data\") pod \"1222e77c-6ae5-4ce1-a15f-caa491260730\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " Oct 30 00:39:08.187981 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.185673 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1222e77c-6ae5-4ce1-a15f-caa491260730-kube-api-access-nqhvf\" (UniqueName: \"kubernetes.io/projected/1222e77c-6ae5-4ce1-a15f-caa491260730-kube-api-access-nqhvf\") pod \"1222e77c-6ae5-4ce1-a15f-caa491260730\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " Oct 30 00:39:08.187981 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.185705 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-istio-envoy\") pod \"1222e77c-6ae5-4ce1-a15f-caa491260730\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " Oct 30 00:39:08.187981 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.185764 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-workload-certs\") pod \"1222e77c-6ae5-4ce1-a15f-caa491260730\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " Oct 30 00:39:08.187981 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.185788 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-credential-socket\") pod \"1222e77c-6ae5-4ce1-a15f-caa491260730\" (UID: \"1222e77c-6ae5-4ce1-a15f-caa491260730\") " Oct 30 00:39:08.187981 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.187373 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1222e77c-6ae5-4ce1-a15f-caa491260730-istiod-ca-cert" pod "1222e77c-6ae5-4ce1-a15f-caa491260730" (UID: "1222e77c-6ae5-4ce1-a15f-caa491260730"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:08.192388 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.191692 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-istio-data" pod "1222e77c-6ae5-4ce1-a15f-caa491260730" (UID: "1222e77c-6ae5-4ce1-a15f-caa491260730"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:08.192388 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.192004 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-workload-socket" pod "1222e77c-6ae5-4ce1-a15f-caa491260730" (UID: "1222e77c-6ae5-4ce1-a15f-caa491260730"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:08.192388 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.192350 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-workload-certs" pod "1222e77c-6ae5-4ce1-a15f-caa491260730" (UID: "1222e77c-6ae5-4ce1-a15f-caa491260730"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:08.195197 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.195159 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-credential-socket" pod "1222e77c-6ae5-4ce1-a15f-caa491260730" (UID: "1222e77c-6ae5-4ce1-a15f-caa491260730"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:08.199009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.198965 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1222e77c-6ae5-4ce1-a15f-caa491260730-kube-api-access-nqhvf" pod "1222e77c-6ae5-4ce1-a15f-caa491260730" (UID: "1222e77c-6ae5-4ce1-a15f-caa491260730"). InnerVolumeSpecName "kube-api-access-nqhvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:08.207718 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.207665 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-istio-envoy" pod "1222e77c-6ae5-4ce1-a15f-caa491260730" (UID: "1222e77c-6ae5-4ce1-a15f-caa491260730"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:08.217134 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.208106 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1222e77c-6ae5-4ce1-a15f-caa491260730-istio-token" pod "1222e77c-6ae5-4ce1-a15f-caa491260730" (UID: "1222e77c-6ae5-4ce1-a15f-caa491260730"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:08.217134 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.217011 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1222e77c-6ae5-4ce1-a15f-caa491260730-istio-podinfo" pod "1222e77c-6ae5-4ce1-a15f-caa491260730" (UID: "1222e77c-6ae5-4ce1-a15f-caa491260730"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:39:08.237614 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.237263 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:39:08.290803 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.290087 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-istio-envoy\") pod \"44355452-926f-4608-b0eb-24bb7e30dcbd\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " Oct 30 00:39:08.291545 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.291515 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44355452-926f-4608-b0eb-24bb7e30dcbd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44355452-926f-4608-b0eb-24bb7e30dcbd-istio-podinfo\") pod \"44355452-926f-4608-b0eb-24bb7e30dcbd\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " Oct 30 00:39:08.291675 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.291590 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44355452-926f-4608-b0eb-24bb7e30dcbd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44355452-926f-4608-b0eb-24bb7e30dcbd-istiod-ca-cert\") pod \"44355452-926f-4608-b0eb-24bb7e30dcbd\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " Oct 30 00:39:08.291733 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.291670 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-credential-socket\") pod \"44355452-926f-4608-b0eb-24bb7e30dcbd\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " Oct 30 00:39:08.291801 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.291782 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-workload-certs\") pod \"44355452-926f-4608-b0eb-24bb7e30dcbd\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " Oct 30 00:39:08.291944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.291893 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44355452-926f-4608-b0eb-24bb7e30dcbd-istio-token\" (UniqueName: \"kubernetes.io/projected/44355452-926f-4608-b0eb-24bb7e30dcbd-istio-token\") pod \"44355452-926f-4608-b0eb-24bb7e30dcbd\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " Oct 30 00:39:08.292046 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.291969 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-workload-socket\") pod \"44355452-926f-4608-b0eb-24bb7e30dcbd\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " Oct 30 00:39:08.292046 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.292016 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-istio-data\") pod \"44355452-926f-4608-b0eb-24bb7e30dcbd\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " Oct 30 00:39:08.292148 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.292080 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44355452-926f-4608-b0eb-24bb7e30dcbd-kube-api-access-pklw7\" (UniqueName: \"kubernetes.io/projected/44355452-926f-4608-b0eb-24bb7e30dcbd-kube-api-access-pklw7\") pod \"44355452-926f-4608-b0eb-24bb7e30dcbd\" (UID: \"44355452-926f-4608-b0eb-24bb7e30dcbd\") " Oct 30 00:39:08.292826 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.292800 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1222e77c-6ae5-4ce1-a15f-caa491260730-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:08.292988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.292834 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:08.292988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.292849 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1222e77c-6ae5-4ce1-a15f-caa491260730-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:08.292988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.292864 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1222e77c-6ae5-4ce1-a15f-caa491260730-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:08.292988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.292876 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:08.292988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.292895 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqhvf\" (UniqueName: \"kubernetes.io/projected/1222e77c-6ae5-4ce1-a15f-caa491260730-kube-api-access-nqhvf\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:08.292988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.292928 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:08.292988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.292944 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:08.292988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.292957 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1222e77c-6ae5-4ce1-a15f-caa491260730-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:08.293821 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.293794 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-credential-socket" pod "44355452-926f-4608-b0eb-24bb7e30dcbd" (UID: "44355452-926f-4608-b0eb-24bb7e30dcbd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:08.295183 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.295157 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-workload-certs" pod "44355452-926f-4608-b0eb-24bb7e30dcbd" (UID: "44355452-926f-4608-b0eb-24bb7e30dcbd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:08.302956 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.301519 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-istio-data" pod "44355452-926f-4608-b0eb-24bb7e30dcbd" (UID: "44355452-926f-4608-b0eb-24bb7e30dcbd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:08.305014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.303221 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-workload-socket" pod "44355452-926f-4608-b0eb-24bb7e30dcbd" (UID: "44355452-926f-4608-b0eb-24bb7e30dcbd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:08.305014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.303248 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44355452-926f-4608-b0eb-24bb7e30dcbd-istiod-ca-cert" pod "44355452-926f-4608-b0eb-24bb7e30dcbd" (UID: "44355452-926f-4608-b0eb-24bb7e30dcbd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:08.306294 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.306259 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44355452-926f-4608-b0eb-24bb7e30dcbd-kube-api-access-pklw7" pod "44355452-926f-4608-b0eb-24bb7e30dcbd" (UID: "44355452-926f-4608-b0eb-24bb7e30dcbd"). InnerVolumeSpecName "kube-api-access-pklw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:08.307894 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.307863 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44355452-926f-4608-b0eb-24bb7e30dcbd-istio-podinfo" pod "44355452-926f-4608-b0eb-24bb7e30dcbd" (UID: "44355452-926f-4608-b0eb-24bb7e30dcbd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:39:08.308424 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.308391 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44355452-926f-4608-b0eb-24bb7e30dcbd-istio-token" pod "44355452-926f-4608-b0eb-24bb7e30dcbd" (UID: "44355452-926f-4608-b0eb-24bb7e30dcbd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:08.308679 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.308652 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-istio-envoy" pod "44355452-926f-4608-b0eb-24bb7e30dcbd" (UID: "44355452-926f-4608-b0eb-24bb7e30dcbd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:08.354879 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.354846 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n"] Oct 30 00:39:08.355171 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.355140 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" podUID="cf513d72-64d7-4444-ae0c-fda3aef69626" containerName="istio-proxy" containerID="cri-o://730b7678f746047c06a2089e3202c50ab0d3ee50106b59e5e0364c215f155197" gracePeriod=30 Oct 30 00:39:08.359460 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.359434 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb"] Oct 30 00:39:08.359664 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.359642 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" podUID="d5dc33b0-82db-44f6-90df-dabe7979e9e6" containerName="istio-proxy" containerID="cri-o://2e6eb98fb28b392bbbf8c3617750e2bb799a1c9e87178a45effe71afa0e6cc64" gracePeriod=30 Oct 30 00:39:08.364019 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.363997 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb"] Oct 30 00:39:08.364165 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.364147 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" podUID="1450166b-4504-4cb0-adac-c43c514a06fd" containerName="istio-proxy" containerID="cri-o://1541d19b136b61c458ea816c5d7b12672dd364fca4ad35a4aef374afd3a448c8" gracePeriod=30 Oct 30 00:39:08.367819 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.367794 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh"] Oct 30 00:39:08.368021 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.367995 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" podUID="6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86" containerName="istio-proxy" containerID="cri-o://08757b8b027232ff366bb669a20944beba02ea04ff260ecb3a73af46602e043c" gracePeriod=30 Oct 30 00:39:08.394370 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.394341 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:08.394370 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.394370 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:08.394556 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.394380 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pklw7\" (UniqueName: \"kubernetes.io/projected/44355452-926f-4608-b0eb-24bb7e30dcbd-kube-api-access-pklw7\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:08.394556 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.394391 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:08.394556 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.394399 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44355452-926f-4608-b0eb-24bb7e30dcbd-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:08.394556 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.394407 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44355452-926f-4608-b0eb-24bb7e30dcbd-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:08.394556 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.394416 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:08.394556 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.394424 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44355452-926f-4608-b0eb-24bb7e30dcbd-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:08.394556 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.394431 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44355452-926f-4608-b0eb-24bb7e30dcbd-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:08.578025 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.577992 5001 generic.go:356] "Generic (PLEG): container finished" podID="1222e77c-6ae5-4ce1-a15f-caa491260730" containerID="6cc703f7001268a7b52c39bea2e3c7a0fbd6c65f5455116e64b37e03d1346ee3" exitCode=0 Oct 30 00:39:08.578234 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.578046 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" event={"ID":"1222e77c-6ae5-4ce1-a15f-caa491260730","Type":"ContainerDied","Data":"6cc703f7001268a7b52c39bea2e3c7a0fbd6c65f5455116e64b37e03d1346ee3"} Oct 30 00:39:08.578234 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.578102 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" event={"ID":"1222e77c-6ae5-4ce1-a15f-caa491260730","Type":"ContainerDied","Data":"d27613a0eb46b85979916787ac5d9994961123d7104be2bc048ebe517f9d158d"} Oct 30 00:39:08.578234 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.578056 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf" Oct 30 00:39:08.578234 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.578131 5001 scope.go:122] "RemoveContainer" containerID="6cc703f7001268a7b52c39bea2e3c7a0fbd6c65f5455116e64b37e03d1346ee3" Oct 30 00:39:08.579680 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.579652 5001 generic.go:356] "Generic (PLEG): container finished" podID="44355452-926f-4608-b0eb-24bb7e30dcbd" containerID="379b761ea39da3c3064a149b8cc39298798c811cf0ad42d8d67857184eedade5" exitCode=0 Oct 30 00:39:08.579773 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.579686 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" event={"ID":"44355452-926f-4608-b0eb-24bb7e30dcbd","Type":"ContainerDied","Data":"379b761ea39da3c3064a149b8cc39298798c811cf0ad42d8d67857184eedade5"} Oct 30 00:39:08.579773 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.579716 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" Oct 30 00:39:08.579773 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.579741 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m" event={"ID":"44355452-926f-4608-b0eb-24bb7e30dcbd","Type":"ContainerDied","Data":"1bada394f033b49e6c1f5166bd4ae2cfd0a6da4a3345b3089cb8d64364f6dbd1"} Oct 30 00:39:08.581544 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.581521 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-05c6b2-55b55d747-dgrp9" Oct 30 00:39:08.581544 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.581520 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-05c6b2-55b55d747-dgrp9" event={"ID":"bd4c0988-ad1a-4288-85f9-573a02594fe5","Type":"ContainerDied","Data":"0b236e3b4ead1bd1351158535dbe817a94668d6adb6f132268e0d5bcc714ed7b"} Oct 30 00:39:08.583710 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.583684 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-05c6b2-6f8558fd8b-kslr7" event={"ID":"63d72c52-189d-4842-8946-d073e53b6879","Type":"ContainerDied","Data":"afaaa7e24d7f4639b50492e2b3e6eaa5041154ea9d31f0abcc00b4ee97585c1a"} Oct 30 00:39:08.583843 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.583781 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-05c6b2-6f8558fd8b-kslr7" Oct 30 00:39:08.597070 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.597051 5001 scope.go:122] "RemoveContainer" containerID="6cc703f7001268a7b52c39bea2e3c7a0fbd6c65f5455116e64b37e03d1346ee3" Oct 30 00:39:08.597372 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:39:08.597352 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cc703f7001268a7b52c39bea2e3c7a0fbd6c65f5455116e64b37e03d1346ee3\": container with ID starting with 6cc703f7001268a7b52c39bea2e3c7a0fbd6c65f5455116e64b37e03d1346ee3 not found: ID does not exist" containerID="6cc703f7001268a7b52c39bea2e3c7a0fbd6c65f5455116e64b37e03d1346ee3" Oct 30 00:39:08.597440 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.597380 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cc703f7001268a7b52c39bea2e3c7a0fbd6c65f5455116e64b37e03d1346ee3"} err="failed to get container status \"6cc703f7001268a7b52c39bea2e3c7a0fbd6c65f5455116e64b37e03d1346ee3\": rpc error: code = NotFound desc = could not find container \"6cc703f7001268a7b52c39bea2e3c7a0fbd6c65f5455116e64b37e03d1346ee3\": container with ID starting with 6cc703f7001268a7b52c39bea2e3c7a0fbd6c65f5455116e64b37e03d1346ee3 not found: ID does not exist" Oct 30 00:39:08.597440 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.597397 5001 scope.go:122] "RemoveContainer" containerID="379b761ea39da3c3064a149b8cc39298798c811cf0ad42d8d67857184eedade5" Oct 30 00:39:08.607544 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.607524 5001 scope.go:122] "RemoveContainer" containerID="379b761ea39da3c3064a149b8cc39298798c811cf0ad42d8d67857184eedade5" Oct 30 00:39:08.607812 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:39:08.607790 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"379b761ea39da3c3064a149b8cc39298798c811cf0ad42d8d67857184eedade5\": container with ID starting with 379b761ea39da3c3064a149b8cc39298798c811cf0ad42d8d67857184eedade5 not found: ID does not exist" containerID="379b761ea39da3c3064a149b8cc39298798c811cf0ad42d8d67857184eedade5" Oct 30 00:39:08.607855 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.607820 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"379b761ea39da3c3064a149b8cc39298798c811cf0ad42d8d67857184eedade5"} err="failed to get container status \"379b761ea39da3c3064a149b8cc39298798c811cf0ad42d8d67857184eedade5\": rpc error: code = NotFound desc = could not find container \"379b761ea39da3c3064a149b8cc39298798c811cf0ad42d8d67857184eedade5\": container with ID starting with 379b761ea39da3c3064a149b8cc39298798c811cf0ad42d8d67857184eedade5 not found: ID does not exist" Oct 30 00:39:08.607855 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.607837 5001 scope.go:122] "RemoveContainer" containerID="0714c29b98a703ee902d129d3a2aae10482bf8d17f7a517136cc518206f53e8c" Oct 30 00:39:08.618315 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.618289 5001 scope.go:122] "RemoveContainer" containerID="30937ffaa0502579eb4c311ae9511628d86d5ca3e656d84ff364d87e32d1443d" Oct 30 00:39:08.664283 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.664260 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-05c6b2-55b55d747-dgrp9"] Oct 30 00:39:08.668579 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.668553 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-05c6b2-55b55d747-dgrp9"] Oct 30 00:39:08.696526 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.696493 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m"] Oct 30 00:39:08.703211 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.703175 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-l4w5m"] Oct 30 00:39:08.731117 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.731076 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-05c6b2-6f8558fd8b-kslr7"] Oct 30 00:39:08.739590 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.739554 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-05c6b2-6f8558fd8b-kslr7"] Oct 30 00:39:08.767759 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.767731 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf"] Oct 30 00:39:08.774039 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.774011 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-m28sf"] Oct 30 00:39:08.942037 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.941958 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1222e77c-6ae5-4ce1-a15f-caa491260730" path="/var/lib/kubelet/pods/1222e77c-6ae5-4ce1-a15f-caa491260730/volumes" Oct 30 00:39:08.942468 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.942453 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44355452-926f-4608-b0eb-24bb7e30dcbd" path="/var/lib/kubelet/pods/44355452-926f-4608-b0eb-24bb7e30dcbd/volumes" Oct 30 00:39:08.942823 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.942810 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63d72c52-189d-4842-8946-d073e53b6879" path="/var/lib/kubelet/pods/63d72c52-189d-4842-8946-d073e53b6879/volumes" Oct 30 00:39:08.943488 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:08.943473 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd4c0988-ad1a-4288-85f9-573a02594fe5" path="/var/lib/kubelet/pods/bd4c0988-ad1a-4288-85f9-573a02594fe5/volumes" Oct 30 00:39:09.039064 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.039023 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-778b5858-jq9cm"] Oct 30 00:39:09.044139 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.044108 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-778b5858-jq9cm"] Oct 30 00:39:09.433197 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.433153 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c93bcf7b-d6xs9"] Oct 30 00:39:09.433730 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.433701 5001 cpu_manager.go:414] "containerMap: removing container" podUID="bd4c0988-ad1a-4288-85f9-573a02594fe5" containerName="payload-pre-processing" Oct 30 00:39:09.433730 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.433723 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd4c0988-ad1a-4288-85f9-573a02594fe5" containerName="payload-pre-processing" Oct 30 00:39:09.433730 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.433734 5001 cpu_manager.go:414] "containerMap: removing container" podUID="be687896-3ef5-4bfc-b170-3e91fcc13a3e" containerName="payload-pre-processing" Oct 30 00:39:09.434011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.433739 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be687896-3ef5-4bfc-b170-3e91fcc13a3e" containerName="payload-pre-processing" Oct 30 00:39:09.434011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.433769 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1222e77c-6ae5-4ce1-a15f-caa491260730" containerName="istio-proxy" Oct 30 00:39:09.434011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.433774 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1222e77c-6ae5-4ce1-a15f-caa491260730" containerName="istio-proxy" Oct 30 00:39:09.434011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.433785 5001 cpu_manager.go:414] "containerMap: removing container" podUID="3f3b775e-8102-4e77-a2f7-272ac5d6ad5a" containerName="payload-processing" Oct 30 00:39:09.434011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.433790 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f3b775e-8102-4e77-a2f7-272ac5d6ad5a" containerName="payload-processing" Oct 30 00:39:09.434011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.433803 5001 cpu_manager.go:414] "containerMap: removing container" podUID="44355452-926f-4608-b0eb-24bb7e30dcbd" containerName="istio-proxy" Oct 30 00:39:09.434011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.433808 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44355452-926f-4608-b0eb-24bb7e30dcbd" containerName="istio-proxy" Oct 30 00:39:09.434011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.433816 5001 cpu_manager.go:414] "containerMap: removing container" podUID="63d72c52-189d-4842-8946-d073e53b6879" containerName="payload-processing" Oct 30 00:39:09.434011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.433820 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63d72c52-189d-4842-8946-d073e53b6879" containerName="payload-processing" Oct 30 00:39:09.434011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.433826 5001 cpu_manager.go:414] "containerMap: removing container" podUID="368f7283-c28c-4ae5-859c-af2488a5c486" containerName="revoke-keys" Oct 30 00:39:09.434011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.433830 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="368f7283-c28c-4ae5-859c-af2488a5c486" containerName="revoke-keys" Oct 30 00:39:09.434011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.433947 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="44355452-926f-4608-b0eb-24bb7e30dcbd" containerName="istio-proxy" Oct 30 00:39:09.434011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.433962 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="63d72c52-189d-4842-8946-d073e53b6879" containerName="payload-processing" Oct 30 00:39:09.434011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.433973 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="368f7283-c28c-4ae5-859c-af2488a5c486" containerName="revoke-keys" Oct 30 00:39:09.434011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.433986 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd4c0988-ad1a-4288-85f9-573a02594fe5" containerName="payload-pre-processing" Oct 30 00:39:09.434011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.433992 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="be687896-3ef5-4bfc-b170-3e91fcc13a3e" containerName="payload-pre-processing" Oct 30 00:39:09.434011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.434004 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1222e77c-6ae5-4ce1-a15f-caa491260730" containerName="istio-proxy" Oct 30 00:39:09.434011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.434011 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f3b775e-8102-4e77-a2f7-272ac5d6ad5a" containerName="payload-processing" Oct 30 00:39:09.439363 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.439338 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93bcf7b-d6xs9" Oct 30 00:39:09.440905 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.440653 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07a30c06-h4vkg"] Oct 30 00:39:09.445629 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.445610 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c93bcf7b-d6xs9"] Oct 30 00:39:09.445745 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.445697 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a30c06-h4vkg" Oct 30 00:39:09.450628 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.450602 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07a30c06-h4vkg"] Oct 30 00:39:09.452049 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.452026 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:39:09.452049 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.452026 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:39:09.452248 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.452142 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:39:09.456339 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.455955 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:39:09.531938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.508165 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8bkm\" (UniqueName: \"kubernetes.io/projected/68744b2f-7594-448c-83f5-c2a3a7574274-kube-api-access-f8bkm\") pod \"maas-api-revoke-sub-07a30c06-h4vkg\" (UID: \"68744b2f-7594-448c-83f5-c2a3a7574274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a30c06-h4vkg" Oct 30 00:39:09.531938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.508299 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6cac301-2c7d-44e7-bad5-1b2a2c280dac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c93bcf7b-d6xs9\" (UID: \"f6cac301-2c7d-44e7-bad5-1b2a2c280dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93bcf7b-d6xs9" Oct 30 00:39:09.531938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.508380 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68744b2f-7594-448c-83f5-c2a3a7574274-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07a30c06-h4vkg\" (UID: \"68744b2f-7594-448c-83f5-c2a3a7574274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a30c06-h4vkg" Oct 30 00:39:09.531938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.508414 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvs69\" (UniqueName: \"kubernetes.io/projected/f6cac301-2c7d-44e7-bad5-1b2a2c280dac-kube-api-access-wvs69\") pod \"maas-api-revoke-sub-c93bcf7b-d6xs9\" (UID: \"f6cac301-2c7d-44e7-bad5-1b2a2c280dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93bcf7b-d6xs9" Oct 30 00:39:09.531938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.508463 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6cac301-2c7d-44e7-bad5-1b2a2c280dac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c93bcf7b-d6xs9\" (UID: \"f6cac301-2c7d-44e7-bad5-1b2a2c280dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93bcf7b-d6xs9" Oct 30 00:39:09.531938 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.508487 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68744b2f-7594-448c-83f5-c2a3a7574274-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07a30c06-h4vkg\" (UID: \"68744b2f-7594-448c-83f5-c2a3a7574274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a30c06-h4vkg" Oct 30 00:39:09.620944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.619628 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6cac301-2c7d-44e7-bad5-1b2a2c280dac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c93bcf7b-d6xs9\" (UID: \"f6cac301-2c7d-44e7-bad5-1b2a2c280dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93bcf7b-d6xs9" Oct 30 00:39:09.620944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.619747 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68744b2f-7594-448c-83f5-c2a3a7574274-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07a30c06-h4vkg\" (UID: \"68744b2f-7594-448c-83f5-c2a3a7574274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a30c06-h4vkg" Oct 30 00:39:09.620944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.619785 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvs69\" (UniqueName: \"kubernetes.io/projected/f6cac301-2c7d-44e7-bad5-1b2a2c280dac-kube-api-access-wvs69\") pod \"maas-api-revoke-sub-c93bcf7b-d6xs9\" (UID: \"f6cac301-2c7d-44e7-bad5-1b2a2c280dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93bcf7b-d6xs9" Oct 30 00:39:09.620944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.619840 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6cac301-2c7d-44e7-bad5-1b2a2c280dac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c93bcf7b-d6xs9\" (UID: \"f6cac301-2c7d-44e7-bad5-1b2a2c280dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93bcf7b-d6xs9" Oct 30 00:39:09.620944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.619867 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68744b2f-7594-448c-83f5-c2a3a7574274-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07a30c06-h4vkg\" (UID: \"68744b2f-7594-448c-83f5-c2a3a7574274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a30c06-h4vkg" Oct 30 00:39:09.620944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.619966 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8bkm\" (UniqueName: \"kubernetes.io/projected/68744b2f-7594-448c-83f5-c2a3a7574274-kube-api-access-f8bkm\") pod \"maas-api-revoke-sub-07a30c06-h4vkg\" (UID: \"68744b2f-7594-448c-83f5-c2a3a7574274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a30c06-h4vkg" Oct 30 00:39:09.625944 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.622944 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68744b2f-7594-448c-83f5-c2a3a7574274-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07a30c06-h4vkg\" (UID: \"68744b2f-7594-448c-83f5-c2a3a7574274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a30c06-h4vkg" Oct 30 00:39:09.639936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.626344 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6cac301-2c7d-44e7-bad5-1b2a2c280dac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c93bcf7b-d6xs9\" (UID: \"f6cac301-2c7d-44e7-bad5-1b2a2c280dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93bcf7b-d6xs9" Oct 30 00:39:09.639936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.632307 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68744b2f-7594-448c-83f5-c2a3a7574274-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07a30c06-h4vkg\" (UID: \"68744b2f-7594-448c-83f5-c2a3a7574274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a30c06-h4vkg" Oct 30 00:39:09.639936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.632654 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6cac301-2c7d-44e7-bad5-1b2a2c280dac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c93bcf7b-d6xs9\" (UID: \"f6cac301-2c7d-44e7-bad5-1b2a2c280dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93bcf7b-d6xs9" Oct 30 00:39:09.641656 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.641623 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8bkm\" (UniqueName: \"kubernetes.io/projected/68744b2f-7594-448c-83f5-c2a3a7574274-kube-api-access-f8bkm\") pod \"maas-api-revoke-sub-07a30c06-h4vkg\" (UID: \"68744b2f-7594-448c-83f5-c2a3a7574274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a30c06-h4vkg" Oct 30 00:39:09.642540 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.642515 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvs69\" (UniqueName: \"kubernetes.io/projected/f6cac301-2c7d-44e7-bad5-1b2a2c280dac-kube-api-access-wvs69\") pod \"maas-api-revoke-sub-c93bcf7b-d6xs9\" (UID: \"f6cac301-2c7d-44e7-bad5-1b2a2c280dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93bcf7b-d6xs9" Oct 30 00:39:09.764088 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.764059 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93bcf7b-d6xs9" Oct 30 00:39:09.773442 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:09.773412 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a30c06-h4vkg" Oct 30 00:39:10.138391 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:10.138366 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c93bcf7b-d6xs9"] Oct 30 00:39:10.139440 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:39:10.139413 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6cac301_2c7d_44e7_bad5_1b2a2c280dac.slice/crio-1616d91c1adf73687a415ec3c626c81bf647a6595482f203ab391787a117f808 WatchSource:0}: Error finding container 1616d91c1adf73687a415ec3c626c81bf647a6595482f203ab391787a117f808: Status 404 returned error can't find the container with id 1616d91c1adf73687a415ec3c626c81bf647a6595482f203ab391787a117f808 Oct 30 00:39:10.153868 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:10.153292 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07a30c06-h4vkg"] Oct 30 00:39:10.161236 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:39:10.161206 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68744b2f_7594_448c_83f5_c2a3a7574274.slice/crio-f0f3987652a26e1c65e097daf372d6d31ca6a706518b36c0ea8c34d406d45476 WatchSource:0}: Error finding container f0f3987652a26e1c65e097daf372d6d31ca6a706518b36c0ea8c34d406d45476: Status 404 returned error can't find the container with id f0f3987652a26e1c65e097daf372d6d31ca6a706518b36c0ea8c34d406d45476 Oct 30 00:39:10.606410 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:10.606358 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93bcf7b-d6xs9" event={"ID":"f6cac301-2c7d-44e7-bad5-1b2a2c280dac","Type":"ContainerStarted","Data":"821faea7e27ca7b59b33b8f202bde64421610e3c4ed68f1928cc2d17e71691e4"} Oct 30 00:39:10.606847 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:10.606436 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93bcf7b-d6xs9" event={"ID":"f6cac301-2c7d-44e7-bad5-1b2a2c280dac","Type":"ContainerStarted","Data":"1616d91c1adf73687a415ec3c626c81bf647a6595482f203ab391787a117f808"} Oct 30 00:39:10.613786 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:10.608374 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a30c06-h4vkg" event={"ID":"68744b2f-7594-448c-83f5-c2a3a7574274","Type":"ContainerStarted","Data":"e5471b8f350e5c90a836f1ec2b54347be525e1c45a3b354fa7639d43c1b049ec"} Oct 30 00:39:10.613786 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:10.608426 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a30c06-h4vkg" event={"ID":"68744b2f-7594-448c-83f5-c2a3a7574274","Type":"ContainerStarted","Data":"f0f3987652a26e1c65e097daf372d6d31ca6a706518b36c0ea8c34d406d45476"} Oct 30 00:39:10.643765 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:10.643714 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93bcf7b-d6xs9" podStartSLOduration=1.643701075 podStartE2EDuration="1.643701075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:39:10.642117716 +0000 UTC m=+2065.847618901" watchObservedRunningTime="2026-10-30 00:39:10.643701075 +0000 UTC m=+2065.849202255" Oct 30 00:39:10.680836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:10.680770 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a30c06-h4vkg" podStartSLOduration=1.680757747 podStartE2EDuration="1.680757747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:39:10.676348588 +0000 UTC m=+2065.881849768" watchObservedRunningTime="2026-10-30 00:39:10.680757747 +0000 UTC m=+2065.886258927" Oct 30 00:39:10.926936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:10.926173 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b492299d-9c08-491b-bcd8-3d4ae73f5522" path="/var/lib/kubelet/pods/b492299d-9c08-491b-bcd8-3d4ae73f5522/volumes" Oct 30 00:39:11.078661 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:11.078618 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-322d20-865f5cc95-vkvd2"] Oct 30 00:39:11.632936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:11.632760 5001 generic.go:356] "Generic (PLEG): container finished" podID="f6cac301-2c7d-44e7-bad5-1b2a2c280dac" containerID="821faea7e27ca7b59b33b8f202bde64421610e3c4ed68f1928cc2d17e71691e4" exitCode=0 Oct 30 00:39:11.632936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:11.632875 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93bcf7b-d6xs9" event={"ID":"f6cac301-2c7d-44e7-bad5-1b2a2c280dac","Type":"ContainerDied","Data":"821faea7e27ca7b59b33b8f202bde64421610e3c4ed68f1928cc2d17e71691e4"} Oct 30 00:39:11.640236 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:11.640204 5001 generic.go:356] "Generic (PLEG): container finished" podID="68744b2f-7594-448c-83f5-c2a3a7574274" containerID="e5471b8f350e5c90a836f1ec2b54347be525e1c45a3b354fa7639d43c1b049ec" exitCode=0 Oct 30 00:39:11.640404 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:11.640294 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a30c06-h4vkg" event={"ID":"68744b2f-7594-448c-83f5-c2a3a7574274","Type":"ContainerDied","Data":"e5471b8f350e5c90a836f1ec2b54347be525e1c45a3b354fa7639d43c1b049ec"} Oct 30 00:39:12.544994 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:12.542216 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t"] Oct 30 00:39:12.548746 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:12.547780 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t" Oct 30 00:39:12.550301 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:12.550276 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t"] Oct 30 00:39:12.566692 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:12.563658 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-7505bd-65fc9799fb-8p95v" podUID="91e5f27d-f718-4239-98ee-b0a17382677e" containerName="payload-processing" containerID="cri-o://5100a7e1faf6eb7eecdc37f3085290ba0e4f6ac1174b4d266e516fd656972caa" gracePeriod=25 Oct 30 00:39:12.570477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:12.570453 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-d96837\"/\"autores-model-d96837-kserve-self-signed-certs\"" Oct 30 00:39:12.570596 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:12.570503 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-d96837\"/\"openshift-service-ca.crt\"" Oct 30 00:39:12.570596 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:12.570541 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-d96837\"/\"default-dockercfg-rprcs\"" Oct 30 00:39:12.570784 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:12.570760 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-d96837\"/\"kube-root-ca.crt\"" Oct 30 00:39:12.570876 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:12.570778 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7505bd-76fc47c6d7-jqkf2" podUID="9306cc6a-b126-4e05-b726-fc9b267a0d78" containerName="payload-pre-processing" containerID="cri-o://3b50ee621aec95eeb34d5f21d11f52f1106a629b91f09552ea4bccbc8b6bff93" gracePeriod=25 Oct 30 00:39:12.669847 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:12.669813 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4vn4\" (UniqueName: \"kubernetes.io/projected/a60854fb-b67c-48f0-b598-99e9e1d8c012-kube-api-access-h4vn4\") pod \"autores-model-d96837-kserve-7d744b4869-ft29t\" (UID: \"a60854fb-b67c-48f0-b598-99e9e1d8c012\") " pod="ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t" Oct 30 00:39:12.670429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:12.669985 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a60854fb-b67c-48f0-b598-99e9e1d8c012-model-cache\") pod \"autores-model-d96837-kserve-7d744b4869-ft29t\" (UID: \"a60854fb-b67c-48f0-b598-99e9e1d8c012\") " pod="ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t" Oct 30 00:39:12.670429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:12.670011 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60854fb-b67c-48f0-b598-99e9e1d8c012-tls-certs\") pod \"autores-model-d96837-kserve-7d744b4869-ft29t\" (UID: \"a60854fb-b67c-48f0-b598-99e9e1d8c012\") " pod="ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t" Oct 30 00:39:12.670429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:12.670113 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a60854fb-b67c-48f0-b598-99e9e1d8c012-tmp-dir\") pod \"autores-model-d96837-kserve-7d744b4869-ft29t\" (UID: \"a60854fb-b67c-48f0-b598-99e9e1d8c012\") " pod="ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t" Oct 30 00:39:12.670429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:12.670146 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a60854fb-b67c-48f0-b598-99e9e1d8c012-home\") pod \"autores-model-d96837-kserve-7d744b4869-ft29t\" (UID: \"a60854fb-b67c-48f0-b598-99e9e1d8c012\") " pod="ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t" Oct 30 00:39:12.670429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:12.670166 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a60854fb-b67c-48f0-b598-99e9e1d8c012-dshm\") pod \"autores-model-d96837-kserve-7d744b4869-ft29t\" (UID: \"a60854fb-b67c-48f0-b598-99e9e1d8c012\") " pod="ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t" Oct 30 00:39:12.778962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:12.778014 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a60854fb-b67c-48f0-b598-99e9e1d8c012-tmp-dir\") pod \"autores-model-d96837-kserve-7d744b4869-ft29t\" (UID: \"a60854fb-b67c-48f0-b598-99e9e1d8c012\") " pod="ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t" Oct 30 00:39:12.778962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:12.778066 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a60854fb-b67c-48f0-b598-99e9e1d8c012-home\") pod \"autores-model-d96837-kserve-7d744b4869-ft29t\" (UID: \"a60854fb-b67c-48f0-b598-99e9e1d8c012\") " pod="ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t" Oct 30 00:39:12.778962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:12.778091 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a60854fb-b67c-48f0-b598-99e9e1d8c012-dshm\") pod \"autores-model-d96837-kserve-7d744b4869-ft29t\" (UID: \"a60854fb-b67c-48f0-b598-99e9e1d8c012\") " pod="ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t" Oct 30 00:39:12.778962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:12.778246 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4vn4\" (UniqueName: \"kubernetes.io/projected/a60854fb-b67c-48f0-b598-99e9e1d8c012-kube-api-access-h4vn4\") pod \"autores-model-d96837-kserve-7d744b4869-ft29t\" (UID: \"a60854fb-b67c-48f0-b598-99e9e1d8c012\") " pod="ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t" Oct 30 00:39:12.778962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:12.778358 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a60854fb-b67c-48f0-b598-99e9e1d8c012-model-cache\") pod \"autores-model-d96837-kserve-7d744b4869-ft29t\" (UID: \"a60854fb-b67c-48f0-b598-99e9e1d8c012\") " pod="ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t" Oct 30 00:39:12.778962 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:12.778383 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60854fb-b67c-48f0-b598-99e9e1d8c012-tls-certs\") pod \"autores-model-d96837-kserve-7d744b4869-ft29t\" (UID: \"a60854fb-b67c-48f0-b598-99e9e1d8c012\") " pod="ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t" Oct 30 00:39:12.829987 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:12.809685 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a60854fb-b67c-48f0-b598-99e9e1d8c012-home\") pod \"autores-model-d96837-kserve-7d744b4869-ft29t\" (UID: \"a60854fb-b67c-48f0-b598-99e9e1d8c012\") " pod="ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t" Oct 30 00:39:12.829987 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:12.821303 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a60854fb-b67c-48f0-b598-99e9e1d8c012-tmp-dir\") pod \"autores-model-d96837-kserve-7d744b4869-ft29t\" (UID: \"a60854fb-b67c-48f0-b598-99e9e1d8c012\") " pod="ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t" Oct 30 00:39:12.829987 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:12.821595 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a60854fb-b67c-48f0-b598-99e9e1d8c012-model-cache\") pod \"autores-model-d96837-kserve-7d744b4869-ft29t\" (UID: \"a60854fb-b67c-48f0-b598-99e9e1d8c012\") " pod="ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t" Oct 30 00:39:12.885083 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:12.876212 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60854fb-b67c-48f0-b598-99e9e1d8c012-tls-certs\") pod \"autores-model-d96837-kserve-7d744b4869-ft29t\" (UID: \"a60854fb-b67c-48f0-b598-99e9e1d8c012\") " pod="ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t" Oct 30 00:39:12.896090 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:12.888359 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4vn4\" (UniqueName: \"kubernetes.io/projected/a60854fb-b67c-48f0-b598-99e9e1d8c012-kube-api-access-h4vn4\") pod \"autores-model-d96837-kserve-7d744b4869-ft29t\" (UID: \"a60854fb-b67c-48f0-b598-99e9e1d8c012\") " pod="ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t" Oct 30 00:39:12.897296 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:12.897208 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a60854fb-b67c-48f0-b598-99e9e1d8c012-dshm\") pod \"autores-model-d96837-kserve-7d744b4869-ft29t\" (UID: \"a60854fb-b67c-48f0-b598-99e9e1d8c012\") " pod="ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t" Oct 30 00:39:13.052839 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.052786 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a30c06-h4vkg" Oct 30 00:39:13.065683 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.065652 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93bcf7b-d6xs9" Oct 30 00:39:13.088834 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.088763 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68744b2f-7594-448c-83f5-c2a3a7574274-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68744b2f-7594-448c-83f5-c2a3a7574274-maas-api-xks-ca\") pod \"68744b2f-7594-448c-83f5-c2a3a7574274\" (UID: \"68744b2f-7594-448c-83f5-c2a3a7574274\") " Oct 30 00:39:13.089011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.088884 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6cac301-2c7d-44e7-bad5-1b2a2c280dac-kube-api-access-wvs69\" (UniqueName: \"kubernetes.io/projected/f6cac301-2c7d-44e7-bad5-1b2a2c280dac-kube-api-access-wvs69\") pod \"f6cac301-2c7d-44e7-bad5-1b2a2c280dac\" (UID: \"f6cac301-2c7d-44e7-bad5-1b2a2c280dac\") " Oct 30 00:39:13.089078 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.089056 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6cac301-2c7d-44e7-bad5-1b2a2c280dac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6cac301-2c7d-44e7-bad5-1b2a2c280dac-maas-api-openshift-ca\") pod \"f6cac301-2c7d-44e7-bad5-1b2a2c280dac\" (UID: \"f6cac301-2c7d-44e7-bad5-1b2a2c280dac\") " Oct 30 00:39:13.089141 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.089096 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68744b2f-7594-448c-83f5-c2a3a7574274-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68744b2f-7594-448c-83f5-c2a3a7574274-maas-api-openshift-ca\") pod \"68744b2f-7594-448c-83f5-c2a3a7574274\" (UID: \"68744b2f-7594-448c-83f5-c2a3a7574274\") " Oct 30 00:39:13.089197 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.089148 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6cac301-2c7d-44e7-bad5-1b2a2c280dac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6cac301-2c7d-44e7-bad5-1b2a2c280dac-maas-api-xks-ca\") pod \"f6cac301-2c7d-44e7-bad5-1b2a2c280dac\" (UID: \"f6cac301-2c7d-44e7-bad5-1b2a2c280dac\") " Oct 30 00:39:13.089197 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.089186 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68744b2f-7594-448c-83f5-c2a3a7574274-kube-api-access-f8bkm\" (UniqueName: \"kubernetes.io/projected/68744b2f-7594-448c-83f5-c2a3a7574274-kube-api-access-f8bkm\") pod \"68744b2f-7594-448c-83f5-c2a3a7574274\" (UID: \"68744b2f-7594-448c-83f5-c2a3a7574274\") " Oct 30 00:39:13.091178 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.089447 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6cac301-2c7d-44e7-bad5-1b2a2c280dac-maas-api-openshift-ca" pod "f6cac301-2c7d-44e7-bad5-1b2a2c280dac" (UID: "f6cac301-2c7d-44e7-bad5-1b2a2c280dac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:13.091178 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.089511 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68744b2f-7594-448c-83f5-c2a3a7574274-maas-api-openshift-ca" pod "68744b2f-7594-448c-83f5-c2a3a7574274" (UID: "68744b2f-7594-448c-83f5-c2a3a7574274"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:13.091178 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.089745 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6cac301-2c7d-44e7-bad5-1b2a2c280dac-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:13.091178 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.089768 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68744b2f-7594-448c-83f5-c2a3a7574274-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:13.091766 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.091733 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68744b2f-7594-448c-83f5-c2a3a7574274-maas-api-xks-ca" pod "68744b2f-7594-448c-83f5-c2a3a7574274" (UID: "68744b2f-7594-448c-83f5-c2a3a7574274"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:39:13.091879 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.091848 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6cac301-2c7d-44e7-bad5-1b2a2c280dac-kube-api-access-wvs69" pod "f6cac301-2c7d-44e7-bad5-1b2a2c280dac" (UID: "f6cac301-2c7d-44e7-bad5-1b2a2c280dac"). InnerVolumeSpecName "kube-api-access-wvs69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:13.091996 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.091930 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68744b2f-7594-448c-83f5-c2a3a7574274-kube-api-access-f8bkm" pod "68744b2f-7594-448c-83f5-c2a3a7574274" (UID: "68744b2f-7594-448c-83f5-c2a3a7574274"). InnerVolumeSpecName "kube-api-access-f8bkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:13.092989 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.092967 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6cac301-2c7d-44e7-bad5-1b2a2c280dac-maas-api-xks-ca" pod "f6cac301-2c7d-44e7-bad5-1b2a2c280dac" (UID: "f6cac301-2c7d-44e7-bad5-1b2a2c280dac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:39:13.168357 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.168321 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t" Oct 30 00:39:13.191261 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.191232 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvs69\" (UniqueName: \"kubernetes.io/projected/f6cac301-2c7d-44e7-bad5-1b2a2c280dac-kube-api-access-wvs69\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:13.191367 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.191263 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6cac301-2c7d-44e7-bad5-1b2a2c280dac-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:13.191367 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.191277 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8bkm\" (UniqueName: \"kubernetes.io/projected/68744b2f-7594-448c-83f5-c2a3a7574274-kube-api-access-f8bkm\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:13.191367 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.191290 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68744b2f-7594-448c-83f5-c2a3a7574274-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:13.205623 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.205371 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:39:13.205623 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.205429 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" podUID="cf513d72-64d7-4444-ae0c-fda3aef69626" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:39:13.505760 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.503714 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t"] Oct 30 00:39:13.664031 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.663827 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7505bd-76fc47c6d7-jqkf2" podUID="9306cc6a-b126-4e05-b726-fc9b267a0d78" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:39:13.664834 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.664770 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-7505bd-65fc9799fb-8p95v" podUID="91e5f27d-f718-4239-98ee-b0a17382677e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:39:13.670525 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.670298 5001 generic.go:356] "Generic (PLEG): container finished" podID="cf513d72-64d7-4444-ae0c-fda3aef69626" containerID="730b7678f746047c06a2089e3202c50ab0d3ee50106b59e5e0364c215f155197" exitCode=0 Oct 30 00:39:13.670525 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.670454 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" event={"ID":"cf513d72-64d7-4444-ae0c-fda3aef69626","Type":"ContainerDied","Data":"730b7678f746047c06a2089e3202c50ab0d3ee50106b59e5e0364c215f155197"} Oct 30 00:39:13.705460 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.705040 5001 generic.go:356] "Generic (PLEG): container finished" podID="6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86" containerID="08757b8b027232ff366bb669a20944beba02ea04ff260ecb3a73af46602e043c" exitCode=0 Oct 30 00:39:13.705460 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.705146 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" event={"ID":"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86","Type":"ContainerDied","Data":"08757b8b027232ff366bb669a20944beba02ea04ff260ecb3a73af46602e043c"} Oct 30 00:39:13.731163 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.713105 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:39:13.731163 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.725266 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93bcf7b-d6xs9" event={"ID":"f6cac301-2c7d-44e7-bad5-1b2a2c280dac","Type":"ContainerDied","Data":"1616d91c1adf73687a415ec3c626c81bf647a6595482f203ab391787a117f808"} Oct 30 00:39:13.731163 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.725334 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1616d91c1adf73687a415ec3c626c81bf647a6595482f203ab391787a117f808" Oct 30 00:39:13.731163 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.725513 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c93bcf7b-d6xs9" Oct 30 00:39:13.746289 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.746066 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a30c06-h4vkg" Oct 30 00:39:13.746289 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.746095 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a30c06-h4vkg" event={"ID":"68744b2f-7594-448c-83f5-c2a3a7574274","Type":"ContainerDied","Data":"f0f3987652a26e1c65e097daf372d6d31ca6a706518b36c0ea8c34d406d45476"} Oct 30 00:39:13.746289 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.746141 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0f3987652a26e1c65e097daf372d6d31ca6a706518b36c0ea8c34d406d45476" Oct 30 00:39:13.765964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.765923 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t" event={"ID":"a60854fb-b67c-48f0-b598-99e9e1d8c012","Type":"ContainerStarted","Data":"7d9875d11b7f8234c9206ad16e39ffc4cc81c425e24cd95b3b4acdcdce30551c"} Oct 30 00:39:13.773469 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.771965 5001 generic.go:356] "Generic (PLEG): container finished" podID="d5dc33b0-82db-44f6-90df-dabe7979e9e6" containerID="2e6eb98fb28b392bbbf8c3617750e2bb799a1c9e87178a45effe71afa0e6cc64" exitCode=0 Oct 30 00:39:13.773469 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.772056 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" event={"ID":"d5dc33b0-82db-44f6-90df-dabe7979e9e6","Type":"ContainerDied","Data":"2e6eb98fb28b392bbbf8c3617750e2bb799a1c9e87178a45effe71afa0e6cc64"} Oct 30 00:39:13.774964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.774888 5001 generic.go:356] "Generic (PLEG): container finished" podID="1450166b-4504-4cb0-adac-c43c514a06fd" containerID="1541d19b136b61c458ea816c5d7b12672dd364fca4ad35a4aef374afd3a448c8" exitCode=0 Oct 30 00:39:13.775079 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.775051 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" event={"ID":"1450166b-4504-4cb0-adac-c43c514a06fd","Type":"ContainerDied","Data":"1541d19b136b61c458ea816c5d7b12672dd364fca4ad35a4aef374afd3a448c8"} Oct 30 00:39:13.783991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.783968 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:39:13.907899 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.907823 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:39:13.941953 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.941902 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istio-token\" (UniqueName: \"kubernetes.io/projected/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istio-token\") pod \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " Oct 30 00:39:13.941953 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.941958 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-istio-data\") pod \"cf513d72-64d7-4444-ae0c-fda3aef69626\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " Oct 30 00:39:13.942208 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.942017 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istio-podinfo\") pod \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " Oct 30 00:39:13.942208 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.942041 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-workload-certs\") pod \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " Oct 30 00:39:13.942208 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.942059 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf513d72-64d7-4444-ae0c-fda3aef69626-kube-api-access-hrlt9\" (UniqueName: \"kubernetes.io/projected/cf513d72-64d7-4444-ae0c-fda3aef69626-kube-api-access-hrlt9\") pod \"cf513d72-64d7-4444-ae0c-fda3aef69626\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " Oct 30 00:39:13.942208 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.942075 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istio-data\") pod \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " Oct 30 00:39:13.942208 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.942108 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-workload-socket\") pod \"cf513d72-64d7-4444-ae0c-fda3aef69626\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " Oct 30 00:39:13.942208 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.942142 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-credential-socket\") pod \"cf513d72-64d7-4444-ae0c-fda3aef69626\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " Oct 30 00:39:13.942208 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.942185 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf513d72-64d7-4444-ae0c-fda3aef69626-istio-token\" (UniqueName: \"kubernetes.io/projected/cf513d72-64d7-4444-ae0c-fda3aef69626-istio-token\") pod \"cf513d72-64d7-4444-ae0c-fda3aef69626\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " Oct 30 00:39:13.942208 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.942203 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istio-envoy\") pod \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " Oct 30 00:39:13.942619 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.942236 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istiod-ca-cert\") pod \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " Oct 30 00:39:13.942619 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.942273 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-kube-api-access-vpzvz\" (UniqueName: \"kubernetes.io/projected/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-kube-api-access-vpzvz\") pod \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " Oct 30 00:39:13.942619 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.942304 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-workload-certs\") pod \"cf513d72-64d7-4444-ae0c-fda3aef69626\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " Oct 30 00:39:13.942619 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.942320 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-credential-socket\") pod \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " Oct 30 00:39:13.942619 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.942340 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-workload-socket\") pod \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\" (UID: \"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86\") " Oct 30 00:39:13.942619 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.942373 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf513d72-64d7-4444-ae0c-fda3aef69626-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf513d72-64d7-4444-ae0c-fda3aef69626-istiod-ca-cert\") pod \"cf513d72-64d7-4444-ae0c-fda3aef69626\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " Oct 30 00:39:13.942619 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.942395 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-istio-envoy\") pod \"cf513d72-64d7-4444-ae0c-fda3aef69626\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " Oct 30 00:39:13.942619 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.942415 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf513d72-64d7-4444-ae0c-fda3aef69626-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf513d72-64d7-4444-ae0c-fda3aef69626-istio-podinfo\") pod \"cf513d72-64d7-4444-ae0c-fda3aef69626\" (UID: \"cf513d72-64d7-4444-ae0c-fda3aef69626\") " Oct 30 00:39:13.943345 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.943310 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-workload-certs" pod "6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86" (UID: "6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:13.943552 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.943495 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-workload-socket" pod "cf513d72-64d7-4444-ae0c-fda3aef69626" (UID: "cf513d72-64d7-4444-ae0c-fda3aef69626"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:13.945504 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.943522 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istio-data" pod "6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86" (UID: "6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:13.945646 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.943713 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-workload-certs" pod "cf513d72-64d7-4444-ae0c-fda3aef69626" (UID: "cf513d72-64d7-4444-ae0c-fda3aef69626"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:13.945767 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.943726 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-credential-socket" pod "cf513d72-64d7-4444-ae0c-fda3aef69626" (UID: "cf513d72-64d7-4444-ae0c-fda3aef69626"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:13.945875 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.943976 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-workload-socket" pod "6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86" (UID: "6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:13.946018 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.944153 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-credential-socket" pod "6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86" (UID: "6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:13.946129 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.944348 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-istio-data" pod "cf513d72-64d7-4444-ae0c-fda3aef69626" (UID: "cf513d72-64d7-4444-ae0c-fda3aef69626"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:13.946246 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.945431 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf513d72-64d7-4444-ae0c-fda3aef69626-istiod-ca-cert" pod "cf513d72-64d7-4444-ae0c-fda3aef69626" (UID: "cf513d72-64d7-4444-ae0c-fda3aef69626"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:13.946362 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.945460 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istiod-ca-cert" pod "6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86" (UID: "6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:13.946595 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.946557 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf513d72-64d7-4444-ae0c-fda3aef69626-istio-token" pod "cf513d72-64d7-4444-ae0c-fda3aef69626" (UID: "cf513d72-64d7-4444-ae0c-fda3aef69626"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:13.947715 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.947648 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf513d72-64d7-4444-ae0c-fda3aef69626-kube-api-access-hrlt9" pod "cf513d72-64d7-4444-ae0c-fda3aef69626" (UID: "cf513d72-64d7-4444-ae0c-fda3aef69626"). InnerVolumeSpecName "kube-api-access-hrlt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:13.947715 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.947683 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-kube-api-access-vpzvz" pod "6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86" (UID: "6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86"). InnerVolumeSpecName "kube-api-access-vpzvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:13.947892 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.947776 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf513d72-64d7-4444-ae0c-fda3aef69626-istio-podinfo" pod "cf513d72-64d7-4444-ae0c-fda3aef69626" (UID: "cf513d72-64d7-4444-ae0c-fda3aef69626"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:39:13.948866 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.948836 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istio-envoy" pod "6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86" (UID: "6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:13.949233 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.949198 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-istio-envoy" pod "cf513d72-64d7-4444-ae0c-fda3aef69626" (UID: "cf513d72-64d7-4444-ae0c-fda3aef69626"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:13.950730 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.950702 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istio-token" pod "6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86" (UID: "6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:13.953760 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.953740 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:39:13.954231 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:13.954204 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istio-podinfo" pod "6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86" (UID: "6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:39:14.045298 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.045261 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-workload-socket\") pod \"1450166b-4504-4cb0-adac-c43c514a06fd\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " Oct 30 00:39:14.045496 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.045323 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1450166b-4504-4cb0-adac-c43c514a06fd-kube-api-access-x8gt8\" (UniqueName: \"kubernetes.io/projected/1450166b-4504-4cb0-adac-c43c514a06fd-kube-api-access-x8gt8\") pod \"1450166b-4504-4cb0-adac-c43c514a06fd\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " Oct 30 00:39:14.045496 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.045354 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-credential-socket\") pod \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " Oct 30 00:39:14.045496 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.045393 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-workload-socket\") pod \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " Oct 30 00:39:14.045496 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.045457 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istio-token\" (UniqueName: \"kubernetes.io/projected/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istio-token\") pod \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " Oct 30 00:39:14.045730 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.045500 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istiod-ca-cert\") pod \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " Oct 30 00:39:14.045730 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.045525 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-workload-socket" pod "1450166b-4504-4cb0-adac-c43c514a06fd" (UID: "1450166b-4504-4cb0-adac-c43c514a06fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:14.045730 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.045552 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istio-envoy\") pod \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " Oct 30 00:39:14.045730 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.045588 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-credential-socket\") pod \"1450166b-4504-4cb0-adac-c43c514a06fd\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " Oct 30 00:39:14.045730 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.045626 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istio-data\") pod \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " Oct 30 00:39:14.045730 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.045662 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1450166b-4504-4cb0-adac-c43c514a06fd-istio-token\" (UniqueName: \"kubernetes.io/projected/1450166b-4504-4cb0-adac-c43c514a06fd-istio-token\") pod \"1450166b-4504-4cb0-adac-c43c514a06fd\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " Oct 30 00:39:14.045730 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.045682 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-workload-socket" pod "d5dc33b0-82db-44f6-90df-dabe7979e9e6" (UID: "d5dc33b0-82db-44f6-90df-dabe7979e9e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:14.045730 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.045695 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-istio-data\") pod \"1450166b-4504-4cb0-adac-c43c514a06fd\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " Oct 30 00:39:14.046278 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.045737 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1450166b-4504-4cb0-adac-c43c514a06fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1450166b-4504-4cb0-adac-c43c514a06fd-istiod-ca-cert\") pod \"1450166b-4504-4cb0-adac-c43c514a06fd\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " Oct 30 00:39:14.046278 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.045781 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-workload-certs\") pod \"1450166b-4504-4cb0-adac-c43c514a06fd\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " Oct 30 00:39:14.046278 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.045815 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5dc33b0-82db-44f6-90df-dabe7979e9e6-kube-api-access-qqnn7\" (UniqueName: \"kubernetes.io/projected/d5dc33b0-82db-44f6-90df-dabe7979e9e6-kube-api-access-qqnn7\") pod \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " Oct 30 00:39:14.046278 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.045845 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-credential-socket" pod "d5dc33b0-82db-44f6-90df-dabe7979e9e6" (UID: "d5dc33b0-82db-44f6-90df-dabe7979e9e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:14.046278 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.045852 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1450166b-4504-4cb0-adac-c43c514a06fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1450166b-4504-4cb0-adac-c43c514a06fd-istio-podinfo\") pod \"1450166b-4504-4cb0-adac-c43c514a06fd\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " Oct 30 00:39:14.046278 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.045891 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-workload-certs\") pod \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " Oct 30 00:39:14.046278 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.045971 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istio-podinfo\") pod \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\" (UID: \"d5dc33b0-82db-44f6-90df-dabe7979e9e6\") " Oct 30 00:39:14.046278 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.046009 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-istio-data" pod "1450166b-4504-4cb0-adac-c43c514a06fd" (UID: "1450166b-4504-4cb0-adac-c43c514a06fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:14.046278 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.046007 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-istio-envoy\") pod \"1450166b-4504-4cb0-adac-c43c514a06fd\" (UID: \"1450166b-4504-4cb0-adac-c43c514a06fd\") " Oct 30 00:39:14.046278 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.046137 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-credential-socket" pod "1450166b-4504-4cb0-adac-c43c514a06fd" (UID: "1450166b-4504-4cb0-adac-c43c514a06fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:14.046824 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.046345 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istiod-ca-cert" pod "d5dc33b0-82db-44f6-90df-dabe7979e9e6" (UID: "d5dc33b0-82db-44f6-90df-dabe7979e9e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:14.046943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.046905 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1450166b-4504-4cb0-adac-c43c514a06fd-istiod-ca-cert" pod "1450166b-4504-4cb0-adac-c43c514a06fd" (UID: "1450166b-4504-4cb0-adac-c43c514a06fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:14.047179 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.047159 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istio-data" pod "d5dc33b0-82db-44f6-90df-dabe7979e9e6" (UID: "d5dc33b0-82db-44f6-90df-dabe7979e9e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:14.047352 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.047335 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-workload-certs" pod "d5dc33b0-82db-44f6-90df-dabe7979e9e6" (UID: "d5dc33b0-82db-44f6-90df-dabe7979e9e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:14.047598 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.047579 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-workload-certs" pod "1450166b-4504-4cb0-adac-c43c514a06fd" (UID: "1450166b-4504-4cb0-adac-c43c514a06fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:14.049223 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.049195 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istio-token" pod "d5dc33b0-82db-44f6-90df-dabe7979e9e6" (UID: "d5dc33b0-82db-44f6-90df-dabe7979e9e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:14.049500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.049480 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.049564 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.049507 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.049564 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.049522 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.049564 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.049541 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrlt9\" (UniqueName: \"kubernetes.io/projected/cf513d72-64d7-4444-ae0c-fda3aef69626-kube-api-access-hrlt9\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.049564 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.049554 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.049695 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.049568 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.049695 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.049581 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.049695 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.049594 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.049695 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.049608 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.049695 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.049621 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf513d72-64d7-4444-ae0c-fda3aef69626-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.049695 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.049634 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.049695 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.049646 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.049695 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.049657 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.049695 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.049670 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpzvz\" (UniqueName: \"kubernetes.io/projected/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-kube-api-access-vpzvz\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.049695 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.049682 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.049695 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.049695 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.050341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.049707 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.050341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.049720 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.050341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.049732 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf513d72-64d7-4444-ae0c-fda3aef69626-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.050341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.049744 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.050341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.049758 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.050341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.049770 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf513d72-64d7-4444-ae0c-fda3aef69626-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.050341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.049782 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.050341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.049794 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.050341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.049806 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1450166b-4504-4cb0-adac-c43c514a06fd-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.050341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.049817 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.050341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.049829 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.050341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.049840 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf513d72-64d7-4444-ae0c-fda3aef69626-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.050341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.049853 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.050341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.049876 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istio-envoy" pod "d5dc33b0-82db-44f6-90df-dabe7979e9e6" (UID: "d5dc33b0-82db-44f6-90df-dabe7979e9e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:14.050341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.050094 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5dc33b0-82db-44f6-90df-dabe7979e9e6-kube-api-access-qqnn7" pod "d5dc33b0-82db-44f6-90df-dabe7979e9e6" (UID: "d5dc33b0-82db-44f6-90df-dabe7979e9e6"). InnerVolumeSpecName "kube-api-access-qqnn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:14.050341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.050159 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istio-podinfo" pod "d5dc33b0-82db-44f6-90df-dabe7979e9e6" (UID: "d5dc33b0-82db-44f6-90df-dabe7979e9e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:39:14.051108 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.050640 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1450166b-4504-4cb0-adac-c43c514a06fd-kube-api-access-x8gt8" pod "1450166b-4504-4cb0-adac-c43c514a06fd" (UID: "1450166b-4504-4cb0-adac-c43c514a06fd"). InnerVolumeSpecName "kube-api-access-x8gt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:14.051108 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.050691 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1450166b-4504-4cb0-adac-c43c514a06fd-istio-token" pod "1450166b-4504-4cb0-adac-c43c514a06fd" (UID: "1450166b-4504-4cb0-adac-c43c514a06fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:14.051108 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.050716 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1450166b-4504-4cb0-adac-c43c514a06fd-istio-podinfo" pod "1450166b-4504-4cb0-adac-c43c514a06fd" (UID: "1450166b-4504-4cb0-adac-c43c514a06fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:39:14.051108 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.050998 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-istio-envoy" pod "1450166b-4504-4cb0-adac-c43c514a06fd" (UID: "1450166b-4504-4cb0-adac-c43c514a06fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:14.150886 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.150851 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.150886 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.150883 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1450166b-4504-4cb0-adac-c43c514a06fd-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.150886 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.150892 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqnn7\" (UniqueName: \"kubernetes.io/projected/d5dc33b0-82db-44f6-90df-dabe7979e9e6-kube-api-access-qqnn7\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.151158 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.150902 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1450166b-4504-4cb0-adac-c43c514a06fd-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.151158 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.150924 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5dc33b0-82db-44f6-90df-dabe7979e9e6-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.151158 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.150936 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1450166b-4504-4cb0-adac-c43c514a06fd-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.151158 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.150944 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8gt8\" (UniqueName: \"kubernetes.io/projected/1450166b-4504-4cb0-adac-c43c514a06fd-kube-api-access-x8gt8\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:14.785838 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.785798 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t" event={"ID":"a60854fb-b67c-48f0-b598-99e9e1d8c012","Type":"ContainerStarted","Data":"f8b2f19a9da8a63f4dfbfa0d937fcf82bfc285f33c51061861063e07b987cf7e"} Oct 30 00:39:14.787418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.787394 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" Oct 30 00:39:14.787535 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.787399 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb" event={"ID":"d5dc33b0-82db-44f6-90df-dabe7979e9e6","Type":"ContainerDied","Data":"64d910a1ccfd17fb994d9009af1245ef3061be2de453b7313391bc352f048e42"} Oct 30 00:39:14.787596 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.787537 5001 scope.go:122] "RemoveContainer" containerID="2e6eb98fb28b392bbbf8c3617750e2bb799a1c9e87178a45effe71afa0e6cc64" Oct 30 00:39:14.789476 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.789449 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" event={"ID":"1450166b-4504-4cb0-adac-c43c514a06fd","Type":"ContainerDied","Data":"630f313b7ad3ed4bc7206d9295f65500cad6ee6eebd0da03b196e07a94178c57"} Oct 30 00:39:14.789612 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.789460 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb" Oct 30 00:39:14.791452 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.791428 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" event={"ID":"cf513d72-64d7-4444-ae0c-fda3aef69626","Type":"ContainerDied","Data":"cea880e918e7cde8e89cd15f6a3ed4d2b6d7f899ad9107da2e124f677d4af536"} Oct 30 00:39:14.791619 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.791420 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n" Oct 30 00:39:14.794333 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.794311 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" Oct 30 00:39:14.794433 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.794318 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh" event={"ID":"6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86","Type":"ContainerDied","Data":"bece2efe3c0f88f83a5afcb317f22ae88098bf0b9c3db0cd25161f5b19ec27af"} Oct 30 00:39:14.812804 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.812781 5001 scope.go:122] "RemoveContainer" containerID="1541d19b136b61c458ea816c5d7b12672dd364fca4ad35a4aef374afd3a448c8" Oct 30 00:39:14.823658 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.823635 5001 scope.go:122] "RemoveContainer" containerID="730b7678f746047c06a2089e3202c50ab0d3ee50106b59e5e0364c215f155197" Oct 30 00:39:14.824765 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.824660 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t" podStartSLOduration=2.824645726 podStartE2EDuration="2.824645726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:39:14.823767097 +0000 UTC m=+2070.029268279" watchObservedRunningTime="2026-10-30 00:39:14.824645726 +0000 UTC m=+2070.030146909" Oct 30 00:39:14.834920 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.834889 5001 scope.go:122] "RemoveContainer" containerID="08757b8b027232ff366bb669a20944beba02ea04ff260ecb3a73af46602e043c" Oct 30 00:39:14.857964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.857905 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb"] Oct 30 00:39:14.861864 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.861835 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-czvzb"] Oct 30 00:39:14.892870 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.892836 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh"] Oct 30 00:39:14.899278 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.899247 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-r9rzh"] Oct 30 00:39:14.921510 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.920358 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1450166b-4504-4cb0-adac-c43c514a06fd" path="/var/lib/kubelet/pods/1450166b-4504-4cb0-adac-c43c514a06fd/volumes" Oct 30 00:39:14.921510 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.921103 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86" path="/var/lib/kubelet/pods/6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86/volumes" Oct 30 00:39:14.924171 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.924147 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb"] Oct 30 00:39:14.929008 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.928982 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-vb4xb"] Oct 30 00:39:14.961257 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.961222 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n"] Oct 30 00:39:14.964794 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:14.964774 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-322d20-openshift-default-7b6dbbcc5f-bqc4n"] Oct 30 00:39:15.169230 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.169149 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t" Oct 30 00:39:15.792991 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.792952 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2e7970f-mnckv"] Oct 30 00:39:15.793796 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.793771 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d5dc33b0-82db-44f6-90df-dabe7979e9e6" containerName="istio-proxy" Oct 30 00:39:15.793852 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.793799 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5dc33b0-82db-44f6-90df-dabe7979e9e6" containerName="istio-proxy" Oct 30 00:39:15.793852 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.793828 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f6cac301-2c7d-44e7-bad5-1b2a2c280dac" containerName="revoke-keys" Oct 30 00:39:15.793852 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.793838 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6cac301-2c7d-44e7-bad5-1b2a2c280dac" containerName="revoke-keys" Oct 30 00:39:15.793976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.793856 5001 cpu_manager.go:414] "containerMap: removing container" podUID="68744b2f-7594-448c-83f5-c2a3a7574274" containerName="revoke-keys" Oct 30 00:39:15.793976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.793864 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68744b2f-7594-448c-83f5-c2a3a7574274" containerName="revoke-keys" Oct 30 00:39:15.793976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.793891 5001 cpu_manager.go:414] "containerMap: removing container" podUID="cf513d72-64d7-4444-ae0c-fda3aef69626" containerName="istio-proxy" Oct 30 00:39:15.793976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.793899 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf513d72-64d7-4444-ae0c-fda3aef69626" containerName="istio-proxy" Oct 30 00:39:15.793976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.793935 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1450166b-4504-4cb0-adac-c43c514a06fd" containerName="istio-proxy" Oct 30 00:39:15.793976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.793943 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1450166b-4504-4cb0-adac-c43c514a06fd" containerName="istio-proxy" Oct 30 00:39:15.793976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.793954 5001 cpu_manager.go:414] "containerMap: removing container" podUID="6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86" containerName="istio-proxy" Oct 30 00:39:15.793976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.793961 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86" containerName="istio-proxy" Oct 30 00:39:15.794215 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.794103 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5dc33b0-82db-44f6-90df-dabe7979e9e6" containerName="istio-proxy" Oct 30 00:39:15.794215 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.794123 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6cac301-2c7d-44e7-bad5-1b2a2c280dac" containerName="revoke-keys" Oct 30 00:39:15.794215 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.794134 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="68744b2f-7594-448c-83f5-c2a3a7574274" containerName="revoke-keys" Oct 30 00:39:15.794215 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.794147 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf513d72-64d7-4444-ae0c-fda3aef69626" containerName="istio-proxy" Oct 30 00:39:15.794215 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.794158 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1450166b-4504-4cb0-adac-c43c514a06fd" containerName="istio-proxy" Oct 30 00:39:15.794215 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.794168 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b9e2b08-e5d3-4ad2-85fd-eda8e00d9f86" containerName="istio-proxy" Oct 30 00:39:15.800266 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.800236 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e7970f-mnckv" Oct 30 00:39:15.817119 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.814043 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab7e1fec-7vhxp"] Oct 30 00:39:15.823168 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.819690 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:39:15.823168 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.820412 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:39:15.823168 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.820476 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:39:15.823168 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.821517 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:39:15.828047 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.828023 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2e7970f-mnckv"] Oct 30 00:39:15.828411 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.828388 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7e1fec-7vhxp" Oct 30 00:39:15.841684 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.841658 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab7e1fec-7vhxp"] Oct 30 00:39:15.868583 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.868544 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtszz\" (UniqueName: \"kubernetes.io/projected/9780e16e-3caf-4dd7-bcba-6bcf4319ef1b-kube-api-access-xtszz\") pod \"maas-api-revoke-sub-f2e7970f-mnckv\" (UID: \"9780e16e-3caf-4dd7-bcba-6bcf4319ef1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e7970f-mnckv" Oct 30 00:39:15.868821 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.868707 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9780e16e-3caf-4dd7-bcba-6bcf4319ef1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2e7970f-mnckv\" (UID: \"9780e16e-3caf-4dd7-bcba-6bcf4319ef1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e7970f-mnckv" Oct 30 00:39:15.868821 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.868753 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrsnl\" (UniqueName: \"kubernetes.io/projected/95d13539-bd4b-4108-ab25-d44712148107-kube-api-access-zrsnl\") pod \"maas-api-revoke-sub-ab7e1fec-7vhxp\" (UID: \"95d13539-bd4b-4108-ab25-d44712148107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7e1fec-7vhxp" Oct 30 00:39:15.868955 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.868871 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95d13539-bd4b-4108-ab25-d44712148107-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab7e1fec-7vhxp\" (UID: \"95d13539-bd4b-4108-ab25-d44712148107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7e1fec-7vhxp" Oct 30 00:39:15.869017 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.868997 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95d13539-bd4b-4108-ab25-d44712148107-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab7e1fec-7vhxp\" (UID: \"95d13539-bd4b-4108-ab25-d44712148107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7e1fec-7vhxp" Oct 30 00:39:15.869061 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.869036 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9780e16e-3caf-4dd7-bcba-6bcf4319ef1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2e7970f-mnckv\" (UID: \"9780e16e-3caf-4dd7-bcba-6bcf4319ef1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e7970f-mnckv" Oct 30 00:39:15.970248 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.970211 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9780e16e-3caf-4dd7-bcba-6bcf4319ef1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2e7970f-mnckv\" (UID: \"9780e16e-3caf-4dd7-bcba-6bcf4319ef1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e7970f-mnckv" Oct 30 00:39:15.970434 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.970258 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrsnl\" (UniqueName: \"kubernetes.io/projected/95d13539-bd4b-4108-ab25-d44712148107-kube-api-access-zrsnl\") pod \"maas-api-revoke-sub-ab7e1fec-7vhxp\" (UID: \"95d13539-bd4b-4108-ab25-d44712148107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7e1fec-7vhxp" Oct 30 00:39:15.970500 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.970436 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95d13539-bd4b-4108-ab25-d44712148107-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab7e1fec-7vhxp\" (UID: \"95d13539-bd4b-4108-ab25-d44712148107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7e1fec-7vhxp" Oct 30 00:39:15.970590 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.970537 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95d13539-bd4b-4108-ab25-d44712148107-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab7e1fec-7vhxp\" (UID: \"95d13539-bd4b-4108-ab25-d44712148107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7e1fec-7vhxp" Oct 30 00:39:15.970590 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.970571 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9780e16e-3caf-4dd7-bcba-6bcf4319ef1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2e7970f-mnckv\" (UID: \"9780e16e-3caf-4dd7-bcba-6bcf4319ef1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e7970f-mnckv" Oct 30 00:39:15.970752 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.970654 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtszz\" (UniqueName: \"kubernetes.io/projected/9780e16e-3caf-4dd7-bcba-6bcf4319ef1b-kube-api-access-xtszz\") pod \"maas-api-revoke-sub-f2e7970f-mnckv\" (UID: \"9780e16e-3caf-4dd7-bcba-6bcf4319ef1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e7970f-mnckv" Oct 30 00:39:15.971328 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.971297 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95d13539-bd4b-4108-ab25-d44712148107-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab7e1fec-7vhxp\" (UID: \"95d13539-bd4b-4108-ab25-d44712148107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7e1fec-7vhxp" Oct 30 00:39:15.971484 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.971311 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9780e16e-3caf-4dd7-bcba-6bcf4319ef1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2e7970f-mnckv\" (UID: \"9780e16e-3caf-4dd7-bcba-6bcf4319ef1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e7970f-mnckv" Oct 30 00:39:15.973102 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.973077 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95d13539-bd4b-4108-ab25-d44712148107-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab7e1fec-7vhxp\" (UID: \"95d13539-bd4b-4108-ab25-d44712148107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7e1fec-7vhxp" Oct 30 00:39:15.973369 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.973344 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9780e16e-3caf-4dd7-bcba-6bcf4319ef1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2e7970f-mnckv\" (UID: \"9780e16e-3caf-4dd7-bcba-6bcf4319ef1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e7970f-mnckv" Oct 30 00:39:15.990662 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.990635 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrsnl\" (UniqueName: \"kubernetes.io/projected/95d13539-bd4b-4108-ab25-d44712148107-kube-api-access-zrsnl\") pod \"maas-api-revoke-sub-ab7e1fec-7vhxp\" (UID: \"95d13539-bd4b-4108-ab25-d44712148107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7e1fec-7vhxp" Oct 30 00:39:15.991060 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:15.991035 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtszz\" (UniqueName: \"kubernetes.io/projected/9780e16e-3caf-4dd7-bcba-6bcf4319ef1b-kube-api-access-xtszz\") pod \"maas-api-revoke-sub-f2e7970f-mnckv\" (UID: \"9780e16e-3caf-4dd7-bcba-6bcf4319ef1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e7970f-mnckv" Oct 30 00:39:16.032121 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:16.032087 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f18e88b3-26p8b"] Oct 30 00:39:16.036418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:16.036393 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cef097b1-f2wkl"] Oct 30 00:39:16.039792 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:16.039772 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cef097b1-f2wkl"] Oct 30 00:39:16.043096 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:16.043045 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f18e88b3-26p8b"] Oct 30 00:39:16.079341 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:16.079306 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-322d20-865f5cc95-vkvd2" podUID="339eea71-a503-460d-b45e-94561c780aca" containerName="payload-pre-processing" containerID="cri-o://93bdcd509b78b5fe9230325d72f45e172efbeded35afbcc622e155077c609fc3" gracePeriod=25 Oct 30 00:39:16.148306 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:16.148268 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e7970f-mnckv" Oct 30 00:39:16.153710 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:16.153687 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7e1fec-7vhxp" Oct 30 00:39:16.478869 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:16.478836 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2e7970f-mnckv"] Oct 30 00:39:16.479555 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:39:16.479523 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9780e16e_3caf_4dd7_bcba_6bcf4319ef1b.slice/crio-93f29cb7ec6c89664bb9191c88cf340b4d38aee7d4ee224da587932fa60aef2f WatchSource:0}: Error finding container 93f29cb7ec6c89664bb9191c88cf340b4d38aee7d4ee224da587932fa60aef2f: Status 404 returned error can't find the container with id 93f29cb7ec6c89664bb9191c88cf340b4d38aee7d4ee224da587932fa60aef2f Oct 30 00:39:16.502435 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:16.502403 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab7e1fec-7vhxp"] Oct 30 00:39:16.503348 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:39:16.503323 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d13539_bd4b_4108_ab25_d44712148107.slice/crio-bdd8da0a3939cc4001fa9af42769da9428d4c3d06beb6042f9157e310c71f8ee WatchSource:0}: Error finding container bdd8da0a3939cc4001fa9af42769da9428d4c3d06beb6042f9157e310c71f8ee: Status 404 returned error can't find the container with id bdd8da0a3939cc4001fa9af42769da9428d4c3d06beb6042f9157e310c71f8ee Oct 30 00:39:16.776284 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:16.776236 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" containerID="cri-o://04f40afe893364fdcd34cb0a4a45118deff02fefee6e0795a66323fcfc996803" gracePeriod=45 Oct 30 00:39:16.839808 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:16.839774 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7e1fec-7vhxp" event={"ID":"95d13539-bd4b-4108-ab25-d44712148107","Type":"ContainerStarted","Data":"bdd8da0a3939cc4001fa9af42769da9428d4c3d06beb6042f9157e310c71f8ee"} Oct 30 00:39:16.841694 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:16.841668 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e7970f-mnckv" event={"ID":"9780e16e-3caf-4dd7-bcba-6bcf4319ef1b","Type":"ContainerStarted","Data":"93f29cb7ec6c89664bb9191c88cf340b4d38aee7d4ee224da587932fa60aef2f"} Oct 30 00:39:16.919733 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:16.919699 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3665be1a-5bbd-49aa-b414-8d168419bc71" path="/var/lib/kubelet/pods/3665be1a-5bbd-49aa-b414-8d168419bc71/volumes" Oct 30 00:39:16.920353 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:16.920257 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf513d72-64d7-4444-ae0c-fda3aef69626" path="/var/lib/kubelet/pods/cf513d72-64d7-4444-ae0c-fda3aef69626/volumes" Oct 30 00:39:16.920925 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:16.920885 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5dc33b0-82db-44f6-90df-dabe7979e9e6" path="/var/lib/kubelet/pods/d5dc33b0-82db-44f6-90df-dabe7979e9e6/volumes" Oct 30 00:39:16.921778 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:16.921755 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d" path="/var/lib/kubelet/pods/f0f8f1fc-97af-4e88-9396-0b9a5f4ddd2d/volumes" Oct 30 00:39:17.063838 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:17.063811 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:39:17.082234 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:17.082204 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef5c38fa-99ee-44ee-a734-793b583df3e4-home\" (UniqueName: \"kubernetes.io/empty-dir/ef5c38fa-99ee-44ee-a734-793b583df3e4-home\") pod \"ef5c38fa-99ee-44ee-a734-793b583df3e4\" (UID: \"ef5c38fa-99ee-44ee-a734-793b583df3e4\") " Oct 30 00:39:17.082388 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:17.082242 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef5c38fa-99ee-44ee-a734-793b583df3e4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef5c38fa-99ee-44ee-a734-793b583df3e4-model-cache\") pod \"ef5c38fa-99ee-44ee-a734-793b583df3e4\" (UID: \"ef5c38fa-99ee-44ee-a734-793b583df3e4\") " Oct 30 00:39:17.082388 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:17.082277 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef5c38fa-99ee-44ee-a734-793b583df3e4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef5c38fa-99ee-44ee-a734-793b583df3e4-tmp-dir\") pod \"ef5c38fa-99ee-44ee-a734-793b583df3e4\" (UID: \"ef5c38fa-99ee-44ee-a734-793b583df3e4\") " Oct 30 00:39:17.082388 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:17.082311 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef5c38fa-99ee-44ee-a734-793b583df3e4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef5c38fa-99ee-44ee-a734-793b583df3e4-dshm\") pod \"ef5c38fa-99ee-44ee-a734-793b583df3e4\" (UID: \"ef5c38fa-99ee-44ee-a734-793b583df3e4\") " Oct 30 00:39:17.082388 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:17.082379 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef5c38fa-99ee-44ee-a734-793b583df3e4-kube-api-access-bvr9d\" (UniqueName: \"kubernetes.io/projected/ef5c38fa-99ee-44ee-a734-793b583df3e4-kube-api-access-bvr9d\") pod \"ef5c38fa-99ee-44ee-a734-793b583df3e4\" (UID: \"ef5c38fa-99ee-44ee-a734-793b583df3e4\") " Oct 30 00:39:17.082653 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:17.082590 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef5c38fa-99ee-44ee-a734-793b583df3e4-tls-certs\" (UniqueName: \"kubernetes.io/secret/ef5c38fa-99ee-44ee-a734-793b583df3e4-tls-certs\") pod \"ef5c38fa-99ee-44ee-a734-793b583df3e4\" (UID: \"ef5c38fa-99ee-44ee-a734-793b583df3e4\") " Oct 30 00:39:17.083178 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:17.083150 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5c38fa-99ee-44ee-a734-793b583df3e4-model-cache" pod "ef5c38fa-99ee-44ee-a734-793b583df3e4" (UID: "ef5c38fa-99ee-44ee-a734-793b583df3e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:17.083573 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:17.083549 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5c38fa-99ee-44ee-a734-793b583df3e4-tmp-dir" pod "ef5c38fa-99ee-44ee-a734-793b583df3e4" (UID: "ef5c38fa-99ee-44ee-a734-793b583df3e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:17.083903 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:17.083877 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5c38fa-99ee-44ee-a734-793b583df3e4-home" pod "ef5c38fa-99ee-44ee-a734-793b583df3e4" (UID: "ef5c38fa-99ee-44ee-a734-793b583df3e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:17.085583 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:17.085558 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5c38fa-99ee-44ee-a734-793b583df3e4-dshm" pod "ef5c38fa-99ee-44ee-a734-793b583df3e4" (UID: "ef5c38fa-99ee-44ee-a734-793b583df3e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:17.086714 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:17.086679 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef5c38fa-99ee-44ee-a734-793b583df3e4-tls-certs" pod "ef5c38fa-99ee-44ee-a734-793b583df3e4" (UID: "ef5c38fa-99ee-44ee-a734-793b583df3e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:39:17.087277 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:17.087251 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef5c38fa-99ee-44ee-a734-793b583df3e4-kube-api-access-bvr9d" pod "ef5c38fa-99ee-44ee-a734-793b583df3e4" (UID: "ef5c38fa-99ee-44ee-a734-793b583df3e4"). InnerVolumeSpecName "kube-api-access-bvr9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:17.184520 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:17.184491 5001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef5c38fa-99ee-44ee-a734-793b583df3e4-tls-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:17.184520 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:17.184516 5001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef5c38fa-99ee-44ee-a734-793b583df3e4-home\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:17.184520 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:17.184525 5001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef5c38fa-99ee-44ee-a734-793b583df3e4-model-cache\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:17.184750 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:17.184533 5001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef5c38fa-99ee-44ee-a734-793b583df3e4-tmp-dir\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:17.184750 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:17.184541 5001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef5c38fa-99ee-44ee-a734-793b583df3e4-dshm\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:17.184750 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:17.184549 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvr9d\" (UniqueName: \"kubernetes.io/projected/ef5c38fa-99ee-44ee-a734-793b583df3e4-kube-api-access-bvr9d\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:17.324274 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:39:17.324228 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:39:17.854356 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:17.854324 5001 generic.go:356] "Generic (PLEG): container finished" podID="95d13539-bd4b-4108-ab25-d44712148107" containerID="0faf6e259cef65583f8e419b56bee28e3dd3ba19c9aa0b29d25968b29a073235" exitCode=0 Oct 30 00:39:17.856876 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:17.854422 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7e1fec-7vhxp" event={"ID":"95d13539-bd4b-4108-ab25-d44712148107","Type":"ContainerDied","Data":"0faf6e259cef65583f8e419b56bee28e3dd3ba19c9aa0b29d25968b29a073235"} Oct 30 00:39:17.856876 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:17.856164 5001 generic.go:356] "Generic (PLEG): container finished" podID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerID="04f40afe893364fdcd34cb0a4a45118deff02fefee6e0795a66323fcfc996803" exitCode=0 Oct 30 00:39:17.856876 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:17.856201 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" Oct 30 00:39:17.856876 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:17.856242 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" event={"ID":"ef5c38fa-99ee-44ee-a734-793b583df3e4","Type":"ContainerDied","Data":"04f40afe893364fdcd34cb0a4a45118deff02fefee6e0795a66323fcfc996803"} Oct 30 00:39:17.856876 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:17.856281 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv" event={"ID":"ef5c38fa-99ee-44ee-a734-793b583df3e4","Type":"ContainerDied","Data":"9cb5625134841c69d5396762b364a297af2b41ab74a89be9c0aafb93a886ffaa"} Oct 30 00:39:17.856876 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:17.856302 5001 scope.go:122] "RemoveContainer" containerID="04f40afe893364fdcd34cb0a4a45118deff02fefee6e0795a66323fcfc996803" Oct 30 00:39:17.858065 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:17.858032 5001 generic.go:356] "Generic (PLEG): container finished" podID="9780e16e-3caf-4dd7-bcba-6bcf4319ef1b" containerID="06817593df5cf5c46864f03a55d0e90603c13c5d2582b9780635a1fbaba4dc91" exitCode=0 Oct 30 00:39:17.858203 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:17.858122 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e7970f-mnckv" event={"ID":"9780e16e-3caf-4dd7-bcba-6bcf4319ef1b","Type":"ContainerDied","Data":"06817593df5cf5c46864f03a55d0e90603c13c5d2582b9780635a1fbaba4dc91"} Oct 30 00:39:17.869456 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:17.869438 5001 scope.go:122] "RemoveContainer" containerID="04f40afe893364fdcd34cb0a4a45118deff02fefee6e0795a66323fcfc996803" Oct 30 00:39:17.869726 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:39:17.869708 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04f40afe893364fdcd34cb0a4a45118deff02fefee6e0795a66323fcfc996803\": container with ID starting with 04f40afe893364fdcd34cb0a4a45118deff02fefee6e0795a66323fcfc996803 not found: ID does not exist" containerID="04f40afe893364fdcd34cb0a4a45118deff02fefee6e0795a66323fcfc996803" Oct 30 00:39:17.869776 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:17.869735 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04f40afe893364fdcd34cb0a4a45118deff02fefee6e0795a66323fcfc996803"} err="failed to get container status \"04f40afe893364fdcd34cb0a4a45118deff02fefee6e0795a66323fcfc996803\": rpc error: code = NotFound desc = could not find container \"04f40afe893364fdcd34cb0a4a45118deff02fefee6e0795a66323fcfc996803\": container with ID starting with 04f40afe893364fdcd34cb0a4a45118deff02fefee6e0795a66323fcfc996803 not found: ID does not exist" Oct 30 00:39:17.922478 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:17.922450 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv"] Oct 30 00:39:17.927659 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:17.927630 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7505bd/simulator-w0-7505bd-kserve-6496f76dc6-gdzsv"] Oct 30 00:39:18.126890 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:18.126798 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-322d20-865f5cc95-vkvd2" podUID="339eea71-a503-460d-b45e-94561c780aca" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:39:18.868203 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:18.868170 5001 generic.go:356] "Generic (PLEG): container finished" podID="6cad4256-1e03-4119-9cbe-4dbe1717600a" containerID="5a1ecd4d86d27be3c301d8922fa4238fdd9622d99a7e18606a26bf98e429130b" exitCode=0 Oct 30 00:39:18.868624 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:18.868244 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" event={"ID":"6cad4256-1e03-4119-9cbe-4dbe1717600a","Type":"ContainerDied","Data":"5a1ecd4d86d27be3c301d8922fa4238fdd9622d99a7e18606a26bf98e429130b"} Oct 30 00:39:18.869006 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:18.868802 5001 scope.go:122] "RemoveContainer" containerID="5a1ecd4d86d27be3c301d8922fa4238fdd9622d99a7e18606a26bf98e429130b" Oct 30 00:39:18.870318 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:18.870286 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:39:18.870424 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:18.870357 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:39:18.870424 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:18.870389 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:39:18.921633 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:18.921603 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" path="/var/lib/kubelet/pods/ef5c38fa-99ee-44ee-a734-793b583df3e4/volumes" Oct 30 00:39:19.070848 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:19.070820 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e7970f-mnckv" Oct 30 00:39:19.112517 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:19.108953 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7e1fec-7vhxp" Oct 30 00:39:19.130072 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:19.128079 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9780e16e-3caf-4dd7-bcba-6bcf4319ef1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9780e16e-3caf-4dd7-bcba-6bcf4319ef1b-maas-api-xks-ca\") pod \"9780e16e-3caf-4dd7-bcba-6bcf4319ef1b\" (UID: \"9780e16e-3caf-4dd7-bcba-6bcf4319ef1b\") " Oct 30 00:39:19.130072 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:19.128182 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9780e16e-3caf-4dd7-bcba-6bcf4319ef1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9780e16e-3caf-4dd7-bcba-6bcf4319ef1b-maas-api-openshift-ca\") pod \"9780e16e-3caf-4dd7-bcba-6bcf4319ef1b\" (UID: \"9780e16e-3caf-4dd7-bcba-6bcf4319ef1b\") " Oct 30 00:39:19.130072 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:19.128316 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9780e16e-3caf-4dd7-bcba-6bcf4319ef1b-kube-api-access-xtszz\" (UniqueName: \"kubernetes.io/projected/9780e16e-3caf-4dd7-bcba-6bcf4319ef1b-kube-api-access-xtszz\") pod \"9780e16e-3caf-4dd7-bcba-6bcf4319ef1b\" (UID: \"9780e16e-3caf-4dd7-bcba-6bcf4319ef1b\") " Oct 30 00:39:19.130072 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:19.129240 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9780e16e-3caf-4dd7-bcba-6bcf4319ef1b-maas-api-openshift-ca" pod "9780e16e-3caf-4dd7-bcba-6bcf4319ef1b" (UID: "9780e16e-3caf-4dd7-bcba-6bcf4319ef1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:19.135679 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:19.135619 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9780e16e-3caf-4dd7-bcba-6bcf4319ef1b-kube-api-access-xtszz" pod "9780e16e-3caf-4dd7-bcba-6bcf4319ef1b" (UID: "9780e16e-3caf-4dd7-bcba-6bcf4319ef1b"). InnerVolumeSpecName "kube-api-access-xtszz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:19.137042 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:19.137012 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9780e16e-3caf-4dd7-bcba-6bcf4319ef1b-maas-api-xks-ca" pod "9780e16e-3caf-4dd7-bcba-6bcf4319ef1b" (UID: "9780e16e-3caf-4dd7-bcba-6bcf4319ef1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:39:19.230104 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:19.230068 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95d13539-bd4b-4108-ab25-d44712148107-kube-api-access-zrsnl\" (UniqueName: \"kubernetes.io/projected/95d13539-bd4b-4108-ab25-d44712148107-kube-api-access-zrsnl\") pod \"95d13539-bd4b-4108-ab25-d44712148107\" (UID: \"95d13539-bd4b-4108-ab25-d44712148107\") " Oct 30 00:39:19.230326 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:19.230124 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95d13539-bd4b-4108-ab25-d44712148107-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95d13539-bd4b-4108-ab25-d44712148107-maas-api-openshift-ca\") pod \"95d13539-bd4b-4108-ab25-d44712148107\" (UID: \"95d13539-bd4b-4108-ab25-d44712148107\") " Oct 30 00:39:19.230326 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:19.230315 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95d13539-bd4b-4108-ab25-d44712148107-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95d13539-bd4b-4108-ab25-d44712148107-maas-api-xks-ca\") pod \"95d13539-bd4b-4108-ab25-d44712148107\" (UID: \"95d13539-bd4b-4108-ab25-d44712148107\") " Oct 30 00:39:19.230650 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:19.230574 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95d13539-bd4b-4108-ab25-d44712148107-maas-api-openshift-ca" pod "95d13539-bd4b-4108-ab25-d44712148107" (UID: "95d13539-bd4b-4108-ab25-d44712148107"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:19.230791 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:19.230698 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9780e16e-3caf-4dd7-bcba-6bcf4319ef1b-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:19.230791 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:19.230724 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9780e16e-3caf-4dd7-bcba-6bcf4319ef1b-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:19.230791 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:19.230739 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtszz\" (UniqueName: \"kubernetes.io/projected/9780e16e-3caf-4dd7-bcba-6bcf4319ef1b-kube-api-access-xtszz\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:19.232549 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:19.232521 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95d13539-bd4b-4108-ab25-d44712148107-maas-api-xks-ca" pod "95d13539-bd4b-4108-ab25-d44712148107" (UID: "95d13539-bd4b-4108-ab25-d44712148107"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:39:19.232638 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:19.232613 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d13539-bd4b-4108-ab25-d44712148107-kube-api-access-zrsnl" pod "95d13539-bd4b-4108-ab25-d44712148107" (UID: "95d13539-bd4b-4108-ab25-d44712148107"). InnerVolumeSpecName "kube-api-access-zrsnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:19.332098 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:19.332059 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95d13539-bd4b-4108-ab25-d44712148107-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:19.332274 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:19.332102 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrsnl\" (UniqueName: \"kubernetes.io/projected/95d13539-bd4b-4108-ab25-d44712148107-kube-api-access-zrsnl\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:19.332274 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:19.332119 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95d13539-bd4b-4108-ab25-d44712148107-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:19.880327 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:19.880296 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" event={"ID":"6cad4256-1e03-4119-9cbe-4dbe1717600a","Type":"ContainerStarted","Data":"5a97e46cc2da747ae8e2780e7af5ca6b57c28a9782bc24b4cd0346e30089a552"} Oct 30 00:39:19.881749 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:19.881726 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7e1fec-7vhxp" Oct 30 00:39:19.881907 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:19.881769 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7e1fec-7vhxp" event={"ID":"95d13539-bd4b-4108-ab25-d44712148107","Type":"ContainerDied","Data":"bdd8da0a3939cc4001fa9af42769da9428d4c3d06beb6042f9157e310c71f8ee"} Oct 30 00:39:19.881907 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:19.881813 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdd8da0a3939cc4001fa9af42769da9428d4c3d06beb6042f9157e310c71f8ee" Oct 30 00:39:19.883160 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:19.883139 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e7970f-mnckv" Oct 30 00:39:19.883160 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:19.883150 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e7970f-mnckv" event={"ID":"9780e16e-3caf-4dd7-bcba-6bcf4319ef1b","Type":"ContainerDied","Data":"93f29cb7ec6c89664bb9191c88cf340b4d38aee7d4ee224da587932fa60aef2f"} Oct 30 00:39:19.883311 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:19.883194 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93f29cb7ec6c89664bb9191c88cf340b4d38aee7d4ee224da587932fa60aef2f" Oct 30 00:39:20.834977 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:20.832261 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:39:20.836307 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:20.836277 5001 patch_prober.go:29] interesting pod/e2e-autores-d96837-openshift-default-7978c98dcc-w445h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body= Oct 30 00:39:20.836453 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:20.836335 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" podUID="6cad4256-1e03-4119-9cbe-4dbe1717600a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" Oct 30 00:39:21.829936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:21.824698 5001 patch_prober.go:29] interesting pod/e2e-autores-d96837-openshift-default-7978c98dcc-w445h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body= Oct 30 00:39:21.829936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:21.824760 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" podUID="6cad4256-1e03-4119-9cbe-4dbe1717600a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" Oct 30 00:39:22.826933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:22.818359 5001 patch_prober.go:29] interesting pod/e2e-autores-d96837-openshift-default-7978c98dcc-w445h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body= Oct 30 00:39:22.826933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:22.818440 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" podUID="6cad4256-1e03-4119-9cbe-4dbe1717600a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" Oct 30 00:39:22.988357 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:22.988324 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7505bd-v5nn2"] Oct 30 00:39:22.993005 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:22.992975 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7505bd-v5nn2"] Oct 30 00:39:23.026934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:23.013003 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:39:23.169437 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:23.169339 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t" Oct 30 00:39:23.182405 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:23.182378 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t" Oct 30 00:39:23.663696 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:23.663656 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-7505bd-65fc9799fb-8p95v" podUID="91e5f27d-f718-4239-98ee-b0a17382677e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:39:23.663897 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:23.663657 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7505bd-76fc47c6d7-jqkf2" podUID="9306cc6a-b126-4e05-b726-fc9b267a0d78" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:39:23.828941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:23.814274 5001 patch_prober.go:29] interesting pod/e2e-autores-d96837-openshift-default-7978c98dcc-w445h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body= Oct 30 00:39:23.828941 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:23.814333 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" podUID="6cad4256-1e03-4119-9cbe-4dbe1717600a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" Oct 30 00:39:23.960396 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:23.960314 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t" Oct 30 00:39:24.813423 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:24.813381 5001 patch_prober.go:29] interesting pod/e2e-autores-d96837-openshift-default-7978c98dcc-w445h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body= Oct 30 00:39:24.813986 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:24.813435 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" podUID="6cad4256-1e03-4119-9cbe-4dbe1717600a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" Oct 30 00:39:24.918001 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:24.917965 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0643035-da6e-4b23-93a6-04cccb0d9692" path="/var/lib/kubelet/pods/a0643035-da6e-4b23-93a6-04cccb0d9692/volumes" Oct 30 00:39:25.818071 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:25.818034 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:39:25.819507 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:25.819485 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:39:27.564208 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:27.564165 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t"] Oct 30 00:39:27.832585 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:27.832496 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d96837-l7s5h"] Oct 30 00:39:27.833122 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:27.833100 5001 cpu_manager.go:414] "containerMap: removing container" podUID="9780e16e-3caf-4dd7-bcba-6bcf4319ef1b" containerName="revoke-keys" Oct 30 00:39:27.833122 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:27.833122 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9780e16e-3caf-4dd7-bcba-6bcf4319ef1b" containerName="revoke-keys" Oct 30 00:39:27.833319 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:27.833151 5001 cpu_manager.go:414] "containerMap: removing container" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" Oct 30 00:39:27.833319 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:27.833159 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" Oct 30 00:39:27.833319 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:27.833202 5001 cpu_manager.go:414] "containerMap: removing container" podUID="95d13539-bd4b-4108-ab25-d44712148107" containerName="revoke-keys" Oct 30 00:39:27.833319 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:27.833210 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95d13539-bd4b-4108-ab25-d44712148107" containerName="revoke-keys" Oct 30 00:39:27.833593 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:27.833346 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="95d13539-bd4b-4108-ab25-d44712148107" containerName="revoke-keys" Oct 30 00:39:27.833593 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:27.833362 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9780e16e-3caf-4dd7-bcba-6bcf4319ef1b" containerName="revoke-keys" Oct 30 00:39:27.833593 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:27.833374 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef5c38fa-99ee-44ee-a734-793b583df3e4" containerName="main" Oct 30 00:39:27.838493 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:27.838471 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d96837-l7s5h" Oct 30 00:39:27.853161 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:27.853135 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:39:27.853161 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:27.853154 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:39:27.853356 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:27.853220 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:39:27.853356 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:27.853228 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:39:27.857943 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:27.857883 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d96837-l7s5h"] Oct 30 00:39:27.937728 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:27.937687 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f94ceedc-64a2-4a09-816f-42ad1b823a34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-d96837-l7s5h\" (UID: \"f94ceedc-64a2-4a09-816f-42ad1b823a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d96837-l7s5h" Oct 30 00:39:27.937904 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:27.937736 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f94ceedc-64a2-4a09-816f-42ad1b823a34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-d96837-l7s5h\" (UID: \"f94ceedc-64a2-4a09-816f-42ad1b823a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d96837-l7s5h" Oct 30 00:39:27.937904 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:27.937864 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tbwt\" (UniqueName: \"kubernetes.io/projected/f94ceedc-64a2-4a09-816f-42ad1b823a34-kube-api-access-5tbwt\") pod \"maas-api-revoke-keys-e2e-autores-d96837-l7s5h\" (UID: \"f94ceedc-64a2-4a09-816f-42ad1b823a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d96837-l7s5h" Oct 30 00:39:28.039520 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:28.039476 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tbwt\" (UniqueName: \"kubernetes.io/projected/f94ceedc-64a2-4a09-816f-42ad1b823a34-kube-api-access-5tbwt\") pod \"maas-api-revoke-keys-e2e-autores-d96837-l7s5h\" (UID: \"f94ceedc-64a2-4a09-816f-42ad1b823a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d96837-l7s5h" Oct 30 00:39:28.039715 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:28.039634 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f94ceedc-64a2-4a09-816f-42ad1b823a34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-d96837-l7s5h\" (UID: \"f94ceedc-64a2-4a09-816f-42ad1b823a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d96837-l7s5h" Oct 30 00:39:28.039715 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:28.039675 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f94ceedc-64a2-4a09-816f-42ad1b823a34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-d96837-l7s5h\" (UID: \"f94ceedc-64a2-4a09-816f-42ad1b823a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d96837-l7s5h" Oct 30 00:39:28.040425 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:28.040399 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f94ceedc-64a2-4a09-816f-42ad1b823a34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-d96837-l7s5h\" (UID: \"f94ceedc-64a2-4a09-816f-42ad1b823a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d96837-l7s5h" Oct 30 00:39:28.042009 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:28.041987 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f94ceedc-64a2-4a09-816f-42ad1b823a34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-d96837-l7s5h\" (UID: \"f94ceedc-64a2-4a09-816f-42ad1b823a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d96837-l7s5h" Oct 30 00:39:28.060061 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:28.060030 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tbwt\" (UniqueName: \"kubernetes.io/projected/f94ceedc-64a2-4a09-816f-42ad1b823a34-kube-api-access-5tbwt\") pod \"maas-api-revoke-keys-e2e-autores-d96837-l7s5h\" (UID: \"f94ceedc-64a2-4a09-816f-42ad1b823a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d96837-l7s5h" Oct 30 00:39:28.127073 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:28.126994 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-322d20-865f5cc95-vkvd2" podUID="339eea71-a503-460d-b45e-94561c780aca" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:39:28.152588 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:28.152555 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d96837-l7s5h" Oct 30 00:39:28.474635 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:39:28.474602 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf94ceedc_64a2_4a09_816f_42ad1b823a34.slice/crio-0e47cd73932a2c1f9e22a04589a58950ee08366eb16fd519226190452fa4e70a WatchSource:0}: Error finding container 0e47cd73932a2c1f9e22a04589a58950ee08366eb16fd519226190452fa4e70a: Status 404 returned error can't find the container with id 0e47cd73932a2c1f9e22a04589a58950ee08366eb16fd519226190452fa4e70a Oct 30 00:39:28.474972 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:28.474949 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d96837-l7s5h"] Oct 30 00:39:29.009631 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:29.009541 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d96837-l7s5h" event={"ID":"f94ceedc-64a2-4a09-816f-42ad1b823a34","Type":"ContainerStarted","Data":"ede6d311c850dc8b9efce79235b045818ce14e455dd09696d7e38d37b9b734f1"} Oct 30 00:39:29.009631 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:29.009608 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d96837-l7s5h" event={"ID":"f94ceedc-64a2-4a09-816f-42ad1b823a34","Type":"ContainerStarted","Data":"0e47cd73932a2c1f9e22a04589a58950ee08366eb16fd519226190452fa4e70a"} Oct 30 00:39:29.047458 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:29.047402 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d96837-l7s5h" podStartSLOduration=2.047387926 podStartE2EDuration="2.047387926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:39:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:39:29.044033921 +0000 UTC m=+2084.249535105" watchObservedRunningTime="2026-10-30 00:39:29.047387926 +0000 UTC m=+2084.252889128" Oct 30 00:39:30.021678 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:30.021639 5001 generic.go:356] "Generic (PLEG): container finished" podID="f94ceedc-64a2-4a09-816f-42ad1b823a34" containerID="ede6d311c850dc8b9efce79235b045818ce14e455dd09696d7e38d37b9b734f1" exitCode=0 Oct 30 00:39:30.022129 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:30.021728 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d96837-l7s5h" event={"ID":"f94ceedc-64a2-4a09-816f-42ad1b823a34","Type":"ContainerDied","Data":"ede6d311c850dc8b9efce79235b045818ce14e455dd09696d7e38d37b9b734f1"} Oct 30 00:39:30.307317 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:30.307234 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-322d20-r8f66"] Oct 30 00:39:30.310856 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:30.310824 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-322d20-r8f66"] Oct 30 00:39:30.893023 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:30.892980 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75"] Oct 30 00:39:30.893247 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:30.893221 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" podUID="4c415517-5484-4905-b9df-8510d1da9ddb" containerName="istio-proxy" containerID="cri-o://d65dbb115be2d9cf05d380376a09bec489fc95b4bdcb9ff3a78dba621359c3ba" gracePeriod=30 Oct 30 00:39:30.918608 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:30.918566 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="368f7283-c28c-4ae5-859c-af2488a5c486" path="/var/lib/kubelet/pods/368f7283-c28c-4ae5-859c-af2488a5c486/volumes" Oct 30 00:39:31.231772 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:31.231745 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d96837-l7s5h" Oct 30 00:39:31.291142 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:31.291111 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f94ceedc-64a2-4a09-816f-42ad1b823a34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f94ceedc-64a2-4a09-816f-42ad1b823a34-maas-api-openshift-ca\") pod \"f94ceedc-64a2-4a09-816f-42ad1b823a34\" (UID: \"f94ceedc-64a2-4a09-816f-42ad1b823a34\") " Oct 30 00:39:31.291142 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:31.291152 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f94ceedc-64a2-4a09-816f-42ad1b823a34-kube-api-access-5tbwt\" (UniqueName: \"kubernetes.io/projected/f94ceedc-64a2-4a09-816f-42ad1b823a34-kube-api-access-5tbwt\") pod \"f94ceedc-64a2-4a09-816f-42ad1b823a34\" (UID: \"f94ceedc-64a2-4a09-816f-42ad1b823a34\") " Oct 30 00:39:31.291403 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:31.291220 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f94ceedc-64a2-4a09-816f-42ad1b823a34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f94ceedc-64a2-4a09-816f-42ad1b823a34-maas-api-xks-ca\") pod \"f94ceedc-64a2-4a09-816f-42ad1b823a34\" (UID: \"f94ceedc-64a2-4a09-816f-42ad1b823a34\") " Oct 30 00:39:31.291557 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:31.291529 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f94ceedc-64a2-4a09-816f-42ad1b823a34-maas-api-openshift-ca" pod "f94ceedc-64a2-4a09-816f-42ad1b823a34" (UID: "f94ceedc-64a2-4a09-816f-42ad1b823a34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:31.293338 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:31.293312 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f94ceedc-64a2-4a09-816f-42ad1b823a34-maas-api-xks-ca" pod "f94ceedc-64a2-4a09-816f-42ad1b823a34" (UID: "f94ceedc-64a2-4a09-816f-42ad1b823a34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:39:31.293464 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:31.293343 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f94ceedc-64a2-4a09-816f-42ad1b823a34-kube-api-access-5tbwt" pod "f94ceedc-64a2-4a09-816f-42ad1b823a34" (UID: "f94ceedc-64a2-4a09-816f-42ad1b823a34"). InnerVolumeSpecName "kube-api-access-5tbwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:31.392497 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:31.392459 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f94ceedc-64a2-4a09-816f-42ad1b823a34-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:31.392497 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:31.392496 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tbwt\" (UniqueName: \"kubernetes.io/projected/f94ceedc-64a2-4a09-816f-42ad1b823a34-kube-api-access-5tbwt\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:31.392712 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:31.392513 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f94ceedc-64a2-4a09-816f-42ad1b823a34-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:32.076840 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:32.076814 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d96837-l7s5h" Oct 30 00:39:32.077031 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:32.076851 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d96837-l7s5h" event={"ID":"f94ceedc-64a2-4a09-816f-42ad1b823a34","Type":"ContainerDied","Data":"0e47cd73932a2c1f9e22a04589a58950ee08366eb16fd519226190452fa4e70a"} Oct 30 00:39:32.077031 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:32.076904 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e47cd73932a2c1f9e22a04589a58950ee08366eb16fd519226190452fa4e70a" Oct 30 00:39:32.562648 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:32.562610 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl"] Oct 30 00:39:32.563035 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:32.562822 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" podUID="27699192-a6f9-4c03-a63d-e19a122dff3a" containerName="istio-proxy" containerID="cri-o://d18662b9f840e4fd76419aec628dc1a54abbc78905d9218d8ce115c3c38a8593" gracePeriod=30 Oct 30 00:39:32.849460 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:32.849441 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7505bd-76fc47c6d7-jqkf2" Oct 30 00:39:32.852467 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:32.852447 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7505bd-65fc9799fb-8p95v" Oct 30 00:39:32.910317 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:32.910285 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91e5f27d-f718-4239-98ee-b0a17382677e-kube-api-access-zcflx\" (UniqueName: \"kubernetes.io/projected/91e5f27d-f718-4239-98ee-b0a17382677e-kube-api-access-zcflx\") pod \"91e5f27d-f718-4239-98ee-b0a17382677e\" (UID: \"91e5f27d-f718-4239-98ee-b0a17382677e\") " Oct 30 00:39:32.910484 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:32.910342 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9306cc6a-b126-4e05-b726-fc9b267a0d78-kube-api-access-wwt87\" (UniqueName: \"kubernetes.io/projected/9306cc6a-b126-4e05-b726-fc9b267a0d78-kube-api-access-wwt87\") pod \"9306cc6a-b126-4e05-b726-fc9b267a0d78\" (UID: \"9306cc6a-b126-4e05-b726-fc9b267a0d78\") " Oct 30 00:39:32.910484 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:32.910390 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9306cc6a-b126-4e05-b726-fc9b267a0d78-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9306cc6a-b126-4e05-b726-fc9b267a0d78-plugins-config-volume\") pod \"9306cc6a-b126-4e05-b726-fc9b267a0d78\" (UID: \"9306cc6a-b126-4e05-b726-fc9b267a0d78\") " Oct 30 00:39:32.910646 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:32.910495 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91e5f27d-f718-4239-98ee-b0a17382677e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e5f27d-f718-4239-98ee-b0a17382677e-plugins-config-volume\") pod \"91e5f27d-f718-4239-98ee-b0a17382677e\" (UID: \"91e5f27d-f718-4239-98ee-b0a17382677e\") " Oct 30 00:39:32.910963 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:32.910932 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91e5f27d-f718-4239-98ee-b0a17382677e-plugins-config-volume" pod "91e5f27d-f718-4239-98ee-b0a17382677e" (UID: "91e5f27d-f718-4239-98ee-b0a17382677e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:32.911349 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:32.911324 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9306cc6a-b126-4e05-b726-fc9b267a0d78-plugins-config-volume" pod "9306cc6a-b126-4e05-b726-fc9b267a0d78" (UID: "9306cc6a-b126-4e05-b726-fc9b267a0d78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:32.912566 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:32.912540 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e5f27d-f718-4239-98ee-b0a17382677e-kube-api-access-zcflx" pod "91e5f27d-f718-4239-98ee-b0a17382677e" (UID: "91e5f27d-f718-4239-98ee-b0a17382677e"). InnerVolumeSpecName "kube-api-access-zcflx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:32.912656 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:32.912581 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9306cc6a-b126-4e05-b726-fc9b267a0d78-kube-api-access-wwt87" pod "9306cc6a-b126-4e05-b726-fc9b267a0d78" (UID: "9306cc6a-b126-4e05-b726-fc9b267a0d78"). InnerVolumeSpecName "kube-api-access-wwt87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:33.011830 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:33.011795 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcflx\" (UniqueName: \"kubernetes.io/projected/91e5f27d-f718-4239-98ee-b0a17382677e-kube-api-access-zcflx\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:33.011830 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:33.011828 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwt87\" (UniqueName: \"kubernetes.io/projected/9306cc6a-b126-4e05-b726-fc9b267a0d78-kube-api-access-wwt87\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:33.011830 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:33.011838 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9306cc6a-b126-4e05-b726-fc9b267a0d78-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:33.012107 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:33.011847 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e5f27d-f718-4239-98ee-b0a17382677e-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:33.086703 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:33.086622 5001 generic.go:356] "Generic (PLEG): container finished" podID="91e5f27d-f718-4239-98ee-b0a17382677e" containerID="5100a7e1faf6eb7eecdc37f3085290ba0e4f6ac1174b4d266e516fd656972caa" exitCode=0 Oct 30 00:39:33.086703 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:33.086678 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7505bd-65fc9799fb-8p95v" Oct 30 00:39:33.086954 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:33.086712 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7505bd-65fc9799fb-8p95v" event={"ID":"91e5f27d-f718-4239-98ee-b0a17382677e","Type":"ContainerDied","Data":"5100a7e1faf6eb7eecdc37f3085290ba0e4f6ac1174b4d266e516fd656972caa"} Oct 30 00:39:33.086954 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:33.086767 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7505bd-65fc9799fb-8p95v" event={"ID":"91e5f27d-f718-4239-98ee-b0a17382677e","Type":"ContainerDied","Data":"9115cfe89cf18da0dfb3305cdd5ea8f95fe819658aa3bd8116847599d093be1f"} Oct 30 00:39:33.086954 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:33.086792 5001 scope.go:122] "RemoveContainer" containerID="5100a7e1faf6eb7eecdc37f3085290ba0e4f6ac1174b4d266e516fd656972caa" Oct 30 00:39:33.088434 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:33.088409 5001 generic.go:356] "Generic (PLEG): container finished" podID="9306cc6a-b126-4e05-b726-fc9b267a0d78" containerID="3b50ee621aec95eeb34d5f21d11f52f1106a629b91f09552ea4bccbc8b6bff93" exitCode=0 Oct 30 00:39:33.088550 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:33.088460 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7505bd-76fc47c6d7-jqkf2" Oct 30 00:39:33.088550 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:33.088461 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7505bd-76fc47c6d7-jqkf2" event={"ID":"9306cc6a-b126-4e05-b726-fc9b267a0d78","Type":"ContainerDied","Data":"3b50ee621aec95eeb34d5f21d11f52f1106a629b91f09552ea4bccbc8b6bff93"} Oct 30 00:39:33.088680 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:33.088587 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7505bd-76fc47c6d7-jqkf2" event={"ID":"9306cc6a-b126-4e05-b726-fc9b267a0d78","Type":"ContainerDied","Data":"67dd33b011cf03f179004a7ea247e0ecb4dd45c5078e8414334b2778c870ee88"} Oct 30 00:39:33.098106 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:33.098089 5001 scope.go:122] "RemoveContainer" containerID="5100a7e1faf6eb7eecdc37f3085290ba0e4f6ac1174b4d266e516fd656972caa" Oct 30 00:39:33.098369 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:39:33.098353 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5100a7e1faf6eb7eecdc37f3085290ba0e4f6ac1174b4d266e516fd656972caa\": container with ID starting with 5100a7e1faf6eb7eecdc37f3085290ba0e4f6ac1174b4d266e516fd656972caa not found: ID does not exist" containerID="5100a7e1faf6eb7eecdc37f3085290ba0e4f6ac1174b4d266e516fd656972caa" Oct 30 00:39:33.098422 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:33.098378 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5100a7e1faf6eb7eecdc37f3085290ba0e4f6ac1174b4d266e516fd656972caa"} err="failed to get container status \"5100a7e1faf6eb7eecdc37f3085290ba0e4f6ac1174b4d266e516fd656972caa\": rpc error: code = NotFound desc = could not find container \"5100a7e1faf6eb7eecdc37f3085290ba0e4f6ac1174b4d266e516fd656972caa\": container with ID starting with 5100a7e1faf6eb7eecdc37f3085290ba0e4f6ac1174b4d266e516fd656972caa not found: ID does not exist" Oct 30 00:39:33.098422 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:33.098393 5001 scope.go:122] "RemoveContainer" containerID="3b50ee621aec95eeb34d5f21d11f52f1106a629b91f09552ea4bccbc8b6bff93" Oct 30 00:39:33.109721 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:33.109704 5001 scope.go:122] "RemoveContainer" containerID="3b50ee621aec95eeb34d5f21d11f52f1106a629b91f09552ea4bccbc8b6bff93" Oct 30 00:39:33.110018 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:39:33.110001 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b50ee621aec95eeb34d5f21d11f52f1106a629b91f09552ea4bccbc8b6bff93\": container with ID starting with 3b50ee621aec95eeb34d5f21d11f52f1106a629b91f09552ea4bccbc8b6bff93 not found: ID does not exist" containerID="3b50ee621aec95eeb34d5f21d11f52f1106a629b91f09552ea4bccbc8b6bff93" Oct 30 00:39:33.110124 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:33.110024 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b50ee621aec95eeb34d5f21d11f52f1106a629b91f09552ea4bccbc8b6bff93"} err="failed to get container status \"3b50ee621aec95eeb34d5f21d11f52f1106a629b91f09552ea4bccbc8b6bff93\": rpc error: code = NotFound desc = could not find container \"3b50ee621aec95eeb34d5f21d11f52f1106a629b91f09552ea4bccbc8b6bff93\": container with ID starting with 3b50ee621aec95eeb34d5f21d11f52f1106a629b91f09552ea4bccbc8b6bff93 not found: ID does not exist" Oct 30 00:39:33.131782 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:33.131761 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7505bd-65fc9799fb-8p95v"] Oct 30 00:39:33.135481 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:33.135460 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7505bd-65fc9799fb-8p95v"] Oct 30 00:39:33.161637 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:33.161616 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7505bd-76fc47c6d7-jqkf2"] Oct 30 00:39:33.165490 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:33.165472 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7505bd-76fc47c6d7-jqkf2"] Oct 30 00:39:34.919553 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:34.919521 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91e5f27d-f718-4239-98ee-b0a17382677e" path="/var/lib/kubelet/pods/91e5f27d-f718-4239-98ee-b0a17382677e/volumes" Oct 30 00:39:34.919958 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:34.919944 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9306cc6a-b126-4e05-b726-fc9b267a0d78" path="/var/lib/kubelet/pods/9306cc6a-b126-4e05-b726-fc9b267a0d78/volumes" Oct 30 00:39:36.122493 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.122466 5001 generic.go:356] "Generic (PLEG): container finished" podID="4c415517-5484-4905-b9df-8510d1da9ddb" containerID="d65dbb115be2d9cf05d380376a09bec489fc95b4bdcb9ff3a78dba621359c3ba" exitCode=0 Oct 30 00:39:36.122930 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.122535 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" event={"ID":"4c415517-5484-4905-b9df-8510d1da9ddb","Type":"ContainerDied","Data":"d65dbb115be2d9cf05d380376a09bec489fc95b4bdcb9ff3a78dba621359c3ba"} Oct 30 00:39:36.209357 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.209325 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:39:36.359952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.359905 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c415517-5484-4905-b9df-8510d1da9ddb-kube-api-access-gvllh\" (UniqueName: \"kubernetes.io/projected/4c415517-5484-4905-b9df-8510d1da9ddb-kube-api-access-gvllh\") pod \"4c415517-5484-4905-b9df-8510d1da9ddb\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " Oct 30 00:39:36.360148 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.359973 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-istio-data\") pod \"4c415517-5484-4905-b9df-8510d1da9ddb\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " Oct 30 00:39:36.360148 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.360023 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c415517-5484-4905-b9df-8510d1da9ddb-istio-token\" (UniqueName: \"kubernetes.io/projected/4c415517-5484-4905-b9df-8510d1da9ddb-istio-token\") pod \"4c415517-5484-4905-b9df-8510d1da9ddb\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " Oct 30 00:39:36.360148 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.360090 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c415517-5484-4905-b9df-8510d1da9ddb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c415517-5484-4905-b9df-8510d1da9ddb-istiod-ca-cert\") pod \"4c415517-5484-4905-b9df-8510d1da9ddb\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " Oct 30 00:39:36.360337 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.360154 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-credential-socket\") pod \"4c415517-5484-4905-b9df-8510d1da9ddb\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " Oct 30 00:39:36.360337 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.360196 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c415517-5484-4905-b9df-8510d1da9ddb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c415517-5484-4905-b9df-8510d1da9ddb-istio-podinfo\") pod \"4c415517-5484-4905-b9df-8510d1da9ddb\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " Oct 30 00:39:36.360337 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.360261 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-istio-envoy\") pod \"4c415517-5484-4905-b9df-8510d1da9ddb\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " Oct 30 00:39:36.360337 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.360297 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-workload-socket\") pod \"4c415517-5484-4905-b9df-8510d1da9ddb\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " Oct 30 00:39:36.360337 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.360324 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-workload-certs\") pod \"4c415517-5484-4905-b9df-8510d1da9ddb\" (UID: \"4c415517-5484-4905-b9df-8510d1da9ddb\") " Oct 30 00:39:36.360692 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.360665 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c415517-5484-4905-b9df-8510d1da9ddb-istiod-ca-cert" pod "4c415517-5484-4905-b9df-8510d1da9ddb" (UID: "4c415517-5484-4905-b9df-8510d1da9ddb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:36.361293 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.360810 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-workload-certs" pod "4c415517-5484-4905-b9df-8510d1da9ddb" (UID: "4c415517-5484-4905-b9df-8510d1da9ddb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:36.361293 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.360938 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-istio-data" pod "4c415517-5484-4905-b9df-8510d1da9ddb" (UID: "4c415517-5484-4905-b9df-8510d1da9ddb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:36.361293 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.361088 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-credential-socket" pod "4c415517-5484-4905-b9df-8510d1da9ddb" (UID: "4c415517-5484-4905-b9df-8510d1da9ddb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:36.361293 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.361140 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.361293 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.361170 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.361293 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.361184 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c415517-5484-4905-b9df-8510d1da9ddb-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.361720 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.361296 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-workload-socket" pod "4c415517-5484-4905-b9df-8510d1da9ddb" (UID: "4c415517-5484-4905-b9df-8510d1da9ddb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:36.363753 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.363726 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c415517-5484-4905-b9df-8510d1da9ddb-kube-api-access-gvllh" pod "4c415517-5484-4905-b9df-8510d1da9ddb" (UID: "4c415517-5484-4905-b9df-8510d1da9ddb"). InnerVolumeSpecName "kube-api-access-gvllh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:36.364070 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.364044 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-istio-envoy" pod "4c415517-5484-4905-b9df-8510d1da9ddb" (UID: "4c415517-5484-4905-b9df-8510d1da9ddb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:36.364180 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.364113 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c415517-5484-4905-b9df-8510d1da9ddb-istio-token" pod "4c415517-5484-4905-b9df-8510d1da9ddb" (UID: "4c415517-5484-4905-b9df-8510d1da9ddb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:36.365084 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.365060 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c415517-5484-4905-b9df-8510d1da9ddb-istio-podinfo" pod "4c415517-5484-4905-b9df-8510d1da9ddb" (UID: "4c415517-5484-4905-b9df-8510d1da9ddb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:39:36.391856 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.391836 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-322d20-865f5cc95-vkvd2" Oct 30 00:39:36.461658 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.461578 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/339eea71-a503-460d-b45e-94561c780aca-kube-api-access-n4k25\" (UniqueName: \"kubernetes.io/projected/339eea71-a503-460d-b45e-94561c780aca-kube-api-access-n4k25\") pod \"339eea71-a503-460d-b45e-94561c780aca\" (UID: \"339eea71-a503-460d-b45e-94561c780aca\") " Oct 30 00:39:36.461810 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.461668 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/339eea71-a503-460d-b45e-94561c780aca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/339eea71-a503-460d-b45e-94561c780aca-plugins-config-volume\") pod \"339eea71-a503-460d-b45e-94561c780aca\" (UID: \"339eea71-a503-460d-b45e-94561c780aca\") " Oct 30 00:39:36.461871 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.461859 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.461926 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.461874 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c415517-5484-4905-b9df-8510d1da9ddb-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.461926 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.461883 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.461926 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.461892 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c415517-5484-4905-b9df-8510d1da9ddb-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.461926 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.461900 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvllh\" (UniqueName: \"kubernetes.io/projected/4c415517-5484-4905-b9df-8510d1da9ddb-kube-api-access-gvllh\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.462062 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.461926 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c415517-5484-4905-b9df-8510d1da9ddb-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.462152 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.462118 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/339eea71-a503-460d-b45e-94561c780aca-plugins-config-volume" pod "339eea71-a503-460d-b45e-94561c780aca" (UID: "339eea71-a503-460d-b45e-94561c780aca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:36.463783 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.463760 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/339eea71-a503-460d-b45e-94561c780aca-kube-api-access-n4k25" pod "339eea71-a503-460d-b45e-94561c780aca" (UID: "339eea71-a503-460d-b45e-94561c780aca"). InnerVolumeSpecName "kube-api-access-n4k25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:36.563236 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.563184 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4k25\" (UniqueName: \"kubernetes.io/projected/339eea71-a503-460d-b45e-94561c780aca-kube-api-access-n4k25\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.563236 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:36.563220 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/339eea71-a503-460d-b45e-94561c780aca-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:37.131253 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:37.131214 5001 generic.go:356] "Generic (PLEG): container finished" podID="339eea71-a503-460d-b45e-94561c780aca" containerID="93bdcd509b78b5fe9230325d72f45e172efbeded35afbcc622e155077c609fc3" exitCode=0 Oct 30 00:39:37.131712 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:37.131281 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-322d20-865f5cc95-vkvd2" Oct 30 00:39:37.131712 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:37.131291 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-322d20-865f5cc95-vkvd2" event={"ID":"339eea71-a503-460d-b45e-94561c780aca","Type":"ContainerDied","Data":"93bdcd509b78b5fe9230325d72f45e172efbeded35afbcc622e155077c609fc3"} Oct 30 00:39:37.131712 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:37.131343 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-322d20-865f5cc95-vkvd2" event={"ID":"339eea71-a503-460d-b45e-94561c780aca","Type":"ContainerDied","Data":"69673fca2fbdbb52d86a6d8dd4d8fd07a7557a733cd269040a22f3d3796942d5"} Oct 30 00:39:37.131712 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:37.131365 5001 scope.go:122] "RemoveContainer" containerID="93bdcd509b78b5fe9230325d72f45e172efbeded35afbcc622e155077c609fc3" Oct 30 00:39:37.133089 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:37.133068 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" event={"ID":"4c415517-5484-4905-b9df-8510d1da9ddb","Type":"ContainerDied","Data":"25abc2bb48d9409bdb2fe9cf34760f424568e8ffda0e20ac9f83e58d37c2d180"} Oct 30 00:39:37.133170 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:37.133090 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75" Oct 30 00:39:37.142322 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:37.142306 5001 scope.go:122] "RemoveContainer" containerID="93bdcd509b78b5fe9230325d72f45e172efbeded35afbcc622e155077c609fc3" Oct 30 00:39:37.142604 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:39:37.142576 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93bdcd509b78b5fe9230325d72f45e172efbeded35afbcc622e155077c609fc3\": container with ID starting with 93bdcd509b78b5fe9230325d72f45e172efbeded35afbcc622e155077c609fc3 not found: ID does not exist" containerID="93bdcd509b78b5fe9230325d72f45e172efbeded35afbcc622e155077c609fc3" Oct 30 00:39:37.142665 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:37.142610 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93bdcd509b78b5fe9230325d72f45e172efbeded35afbcc622e155077c609fc3"} err="failed to get container status \"93bdcd509b78b5fe9230325d72f45e172efbeded35afbcc622e155077c609fc3\": rpc error: code = NotFound desc = could not find container \"93bdcd509b78b5fe9230325d72f45e172efbeded35afbcc622e155077c609fc3\": container with ID starting with 93bdcd509b78b5fe9230325d72f45e172efbeded35afbcc622e155077c609fc3 not found: ID does not exist" Oct 30 00:39:37.142665 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:37.142627 5001 scope.go:122] "RemoveContainer" containerID="d65dbb115be2d9cf05d380376a09bec489fc95b4bdcb9ff3a78dba621359c3ba" Oct 30 00:39:37.177019 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:37.176997 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75"] Oct 30 00:39:37.186840 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:37.186817 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7505bd-openshift-default-5ddd594bc9-wmr75"] Oct 30 00:39:37.210738 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:37.210712 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-322d20-865f5cc95-vkvd2"] Oct 30 00:39:37.215785 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:37.215759 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-322d20-865f5cc95-vkvd2"] Oct 30 00:39:37.829759 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:37.829702 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:39:37.980202 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:37.977677 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-istio-data\") pod \"27699192-a6f9-4c03-a63d-e19a122dff3a\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " Oct 30 00:39:37.980202 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:37.977727 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27699192-a6f9-4c03-a63d-e19a122dff3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27699192-a6f9-4c03-a63d-e19a122dff3a-istiod-ca-cert\") pod \"27699192-a6f9-4c03-a63d-e19a122dff3a\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " Oct 30 00:39:37.980202 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:37.977846 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27699192-a6f9-4c03-a63d-e19a122dff3a-istio-token\" (UniqueName: \"kubernetes.io/projected/27699192-a6f9-4c03-a63d-e19a122dff3a-istio-token\") pod \"27699192-a6f9-4c03-a63d-e19a122dff3a\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " Oct 30 00:39:37.980202 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:37.977899 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27699192-a6f9-4c03-a63d-e19a122dff3a-kube-api-access-pplkb\" (UniqueName: \"kubernetes.io/projected/27699192-a6f9-4c03-a63d-e19a122dff3a-kube-api-access-pplkb\") pod \"27699192-a6f9-4c03-a63d-e19a122dff3a\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " Oct 30 00:39:37.980202 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:37.978028 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27699192-a6f9-4c03-a63d-e19a122dff3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27699192-a6f9-4c03-a63d-e19a122dff3a-istio-podinfo\") pod \"27699192-a6f9-4c03-a63d-e19a122dff3a\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " Oct 30 00:39:37.980202 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:37.978089 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-istio-envoy\") pod \"27699192-a6f9-4c03-a63d-e19a122dff3a\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " Oct 30 00:39:37.980202 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:37.978146 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-credential-socket\") pod \"27699192-a6f9-4c03-a63d-e19a122dff3a\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " Oct 30 00:39:37.980202 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:37.978244 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-workload-certs\") pod \"27699192-a6f9-4c03-a63d-e19a122dff3a\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " Oct 30 00:39:37.980851 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:37.980218 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-workload-socket\") pod \"27699192-a6f9-4c03-a63d-e19a122dff3a\" (UID: \"27699192-a6f9-4c03-a63d-e19a122dff3a\") " Oct 30 00:39:37.981343 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:37.979687 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27699192-a6f9-4c03-a63d-e19a122dff3a-istiod-ca-cert" pod "27699192-a6f9-4c03-a63d-e19a122dff3a" (UID: "27699192-a6f9-4c03-a63d-e19a122dff3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:37.981530 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:37.979980 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-istio-data" pod "27699192-a6f9-4c03-a63d-e19a122dff3a" (UID: "27699192-a6f9-4c03-a63d-e19a122dff3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:37.981530 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:37.980588 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-credential-socket" pod "27699192-a6f9-4c03-a63d-e19a122dff3a" (UID: "27699192-a6f9-4c03-a63d-e19a122dff3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:37.981530 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:37.981099 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-workload-socket" pod "27699192-a6f9-4c03-a63d-e19a122dff3a" (UID: "27699192-a6f9-4c03-a63d-e19a122dff3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:37.981530 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:37.981315 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-workload-certs" pod "27699192-a6f9-4c03-a63d-e19a122dff3a" (UID: "27699192-a6f9-4c03-a63d-e19a122dff3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:37.983386 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:37.983357 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27699192-a6f9-4c03-a63d-e19a122dff3a-istio-token" pod "27699192-a6f9-4c03-a63d-e19a122dff3a" (UID: "27699192-a6f9-4c03-a63d-e19a122dff3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:37.984177 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:37.984156 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-istio-envoy" pod "27699192-a6f9-4c03-a63d-e19a122dff3a" (UID: "27699192-a6f9-4c03-a63d-e19a122dff3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:37.984440 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:37.984417 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27699192-a6f9-4c03-a63d-e19a122dff3a-kube-api-access-pplkb" pod "27699192-a6f9-4c03-a63d-e19a122dff3a" (UID: "27699192-a6f9-4c03-a63d-e19a122dff3a"). InnerVolumeSpecName "kube-api-access-pplkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:37.984707 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:37.984688 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27699192-a6f9-4c03-a63d-e19a122dff3a-istio-podinfo" pod "27699192-a6f9-4c03-a63d-e19a122dff3a" (UID: "27699192-a6f9-4c03-a63d-e19a122dff3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:39:38.082478 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:38.082380 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27699192-a6f9-4c03-a63d-e19a122dff3a-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:38.082478 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:38.082418 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27699192-a6f9-4c03-a63d-e19a122dff3a-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:38.082478 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:38.082433 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pplkb\" (UniqueName: \"kubernetes.io/projected/27699192-a6f9-4c03-a63d-e19a122dff3a-kube-api-access-pplkb\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:38.082478 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:38.082448 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27699192-a6f9-4c03-a63d-e19a122dff3a-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:38.082478 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:38.082464 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:38.082478 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:38.082478 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:38.082952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:38.082491 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:38.082952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:38.082506 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:38.082952 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:38.082517 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27699192-a6f9-4c03-a63d-e19a122dff3a-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:38.145263 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:38.145230 5001 generic.go:356] "Generic (PLEG): container finished" podID="27699192-a6f9-4c03-a63d-e19a122dff3a" containerID="d18662b9f840e4fd76419aec628dc1a54abbc78905d9218d8ce115c3c38a8593" exitCode=0 Oct 30 00:39:38.145771 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:38.145281 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" Oct 30 00:39:38.145771 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:38.145319 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" event={"ID":"27699192-a6f9-4c03-a63d-e19a122dff3a","Type":"ContainerDied","Data":"d18662b9f840e4fd76419aec628dc1a54abbc78905d9218d8ce115c3c38a8593"} Oct 30 00:39:38.145771 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:38.145381 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl" event={"ID":"27699192-a6f9-4c03-a63d-e19a122dff3a","Type":"ContainerDied","Data":"88ccee7ae347a9a1bdef5dfb73a1b385bed12e575a539eac4232af804026366f"} Oct 30 00:39:38.145771 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:38.145414 5001 scope.go:122] "RemoveContainer" containerID="d18662b9f840e4fd76419aec628dc1a54abbc78905d9218d8ce115c3c38a8593" Oct 30 00:39:38.157833 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:38.157811 5001 scope.go:122] "RemoveContainer" containerID="d18662b9f840e4fd76419aec628dc1a54abbc78905d9218d8ce115c3c38a8593" Oct 30 00:39:38.158224 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:39:38.158198 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d18662b9f840e4fd76419aec628dc1a54abbc78905d9218d8ce115c3c38a8593\": container with ID starting with d18662b9f840e4fd76419aec628dc1a54abbc78905d9218d8ce115c3c38a8593 not found: ID does not exist" containerID="d18662b9f840e4fd76419aec628dc1a54abbc78905d9218d8ce115c3c38a8593" Oct 30 00:39:38.158305 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:38.158232 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d18662b9f840e4fd76419aec628dc1a54abbc78905d9218d8ce115c3c38a8593"} err="failed to get container status \"d18662b9f840e4fd76419aec628dc1a54abbc78905d9218d8ce115c3c38a8593\": rpc error: code = NotFound desc = could not find container \"d18662b9f840e4fd76419aec628dc1a54abbc78905d9218d8ce115c3c38a8593\": container with ID starting with d18662b9f840e4fd76419aec628dc1a54abbc78905d9218d8ce115c3c38a8593 not found: ID does not exist" Oct 30 00:39:38.204122 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:38.204088 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl"] Oct 30 00:39:38.212887 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:38.212849 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-9llzl"] Oct 30 00:39:38.303176 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:38.303127 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-d96837-5565b8fc78-qwjl5"] Oct 30 00:39:38.322649 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:38.322608 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d96837-55b85d6d8c-tgmp9"] Oct 30 00:39:38.917512 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:38.917482 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27699192-a6f9-4c03-a63d-e19a122dff3a" path="/var/lib/kubelet/pods/27699192-a6f9-4c03-a63d-e19a122dff3a/volumes" Oct 30 00:39:38.917872 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:38.917859 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="339eea71-a503-460d-b45e-94561c780aca" path="/var/lib/kubelet/pods/339eea71-a503-460d-b45e-94561c780aca/volumes" Oct 30 00:39:38.918292 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:38.918279 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c415517-5484-4905-b9df-8510d1da9ddb" path="/var/lib/kubelet/pods/4c415517-5484-4905-b9df-8510d1da9ddb/volumes" Oct 30 00:39:39.744646 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:39.744611 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d96837-l7s5h"] Oct 30 00:39:39.754296 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:39.754267 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d96837-l7s5h"] Oct 30 00:39:40.920587 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:40.920554 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f94ceedc-64a2-4a09-816f-42ad1b823a34" path="/var/lib/kubelet/pods/f94ceedc-64a2-4a09-816f-42ad1b823a34/volumes" Oct 30 00:39:42.587388 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:42.587334 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t" podUID="a60854fb-b67c-48f0-b598-99e9e1d8c012" containerName="main" containerID="cri-o://f8b2f19a9da8a63f4dfbfa0d937fcf82bfc285f33c51061861063e07b987cf7e" gracePeriod=45 Oct 30 00:39:42.911336 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:42.911302 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t" Oct 30 00:39:43.034806 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.034760 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a60854fb-b67c-48f0-b598-99e9e1d8c012-home\" (UniqueName: \"kubernetes.io/empty-dir/a60854fb-b67c-48f0-b598-99e9e1d8c012-home\") pod \"a60854fb-b67c-48f0-b598-99e9e1d8c012\" (UID: \"a60854fb-b67c-48f0-b598-99e9e1d8c012\") " Oct 30 00:39:43.035065 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.034831 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a60854fb-b67c-48f0-b598-99e9e1d8c012-tls-certs\" (UniqueName: \"kubernetes.io/secret/a60854fb-b67c-48f0-b598-99e9e1d8c012-tls-certs\") pod \"a60854fb-b67c-48f0-b598-99e9e1d8c012\" (UID: \"a60854fb-b67c-48f0-b598-99e9e1d8c012\") " Oct 30 00:39:43.035065 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.034865 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a60854fb-b67c-48f0-b598-99e9e1d8c012-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a60854fb-b67c-48f0-b598-99e9e1d8c012-tmp-dir\") pod \"a60854fb-b67c-48f0-b598-99e9e1d8c012\" (UID: \"a60854fb-b67c-48f0-b598-99e9e1d8c012\") " Oct 30 00:39:43.035065 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.034900 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a60854fb-b67c-48f0-b598-99e9e1d8c012-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a60854fb-b67c-48f0-b598-99e9e1d8c012-dshm\") pod \"a60854fb-b67c-48f0-b598-99e9e1d8c012\" (UID: \"a60854fb-b67c-48f0-b598-99e9e1d8c012\") " Oct 30 00:39:43.035065 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.034973 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a60854fb-b67c-48f0-b598-99e9e1d8c012-kube-api-access-h4vn4\" (UniqueName: \"kubernetes.io/projected/a60854fb-b67c-48f0-b598-99e9e1d8c012-kube-api-access-h4vn4\") pod \"a60854fb-b67c-48f0-b598-99e9e1d8c012\" (UID: \"a60854fb-b67c-48f0-b598-99e9e1d8c012\") " Oct 30 00:39:43.035065 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.035006 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a60854fb-b67c-48f0-b598-99e9e1d8c012-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a60854fb-b67c-48f0-b598-99e9e1d8c012-model-cache\") pod \"a60854fb-b67c-48f0-b598-99e9e1d8c012\" (UID: \"a60854fb-b67c-48f0-b598-99e9e1d8c012\") " Oct 30 00:39:43.035362 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.035125 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a60854fb-b67c-48f0-b598-99e9e1d8c012-home" pod "a60854fb-b67c-48f0-b598-99e9e1d8c012" (UID: "a60854fb-b67c-48f0-b598-99e9e1d8c012"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:43.035362 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.035174 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a60854fb-b67c-48f0-b598-99e9e1d8c012-tmp-dir" pod "a60854fb-b67c-48f0-b598-99e9e1d8c012" (UID: "a60854fb-b67c-48f0-b598-99e9e1d8c012"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:43.035362 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.035345 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a60854fb-b67c-48f0-b598-99e9e1d8c012-model-cache" pod "a60854fb-b67c-48f0-b598-99e9e1d8c012" (UID: "a60854fb-b67c-48f0-b598-99e9e1d8c012"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:43.035630 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.035608 5001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a60854fb-b67c-48f0-b598-99e9e1d8c012-model-cache\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:43.035709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.035634 5001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a60854fb-b67c-48f0-b598-99e9e1d8c012-home\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:43.035709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.035652 5001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a60854fb-b67c-48f0-b598-99e9e1d8c012-tmp-dir\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:43.037221 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.037199 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a60854fb-b67c-48f0-b598-99e9e1d8c012-dshm" pod "a60854fb-b67c-48f0-b598-99e9e1d8c012" (UID: "a60854fb-b67c-48f0-b598-99e9e1d8c012"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:43.037312 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.037237 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a60854fb-b67c-48f0-b598-99e9e1d8c012-tls-certs" pod "a60854fb-b67c-48f0-b598-99e9e1d8c012" (UID: "a60854fb-b67c-48f0-b598-99e9e1d8c012"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:39:43.037361 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.037321 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a60854fb-b67c-48f0-b598-99e9e1d8c012-kube-api-access-h4vn4" pod "a60854fb-b67c-48f0-b598-99e9e1d8c012" (UID: "a60854fb-b67c-48f0-b598-99e9e1d8c012"). InnerVolumeSpecName "kube-api-access-h4vn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:43.136509 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.136419 5001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60854fb-b67c-48f0-b598-99e9e1d8c012-tls-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:43.136509 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.136460 5001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a60854fb-b67c-48f0-b598-99e9e1d8c012-dshm\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:43.136509 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.136470 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4vn4\" (UniqueName: \"kubernetes.io/projected/a60854fb-b67c-48f0-b598-99e9e1d8c012-kube-api-access-h4vn4\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:43.191300 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.191265 5001 generic.go:356] "Generic (PLEG): container finished" podID="a60854fb-b67c-48f0-b598-99e9e1d8c012" containerID="f8b2f19a9da8a63f4dfbfa0d937fcf82bfc285f33c51061861063e07b987cf7e" exitCode=0 Oct 30 00:39:43.191462 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.191306 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t" Oct 30 00:39:43.191462 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.191311 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t" event={"ID":"a60854fb-b67c-48f0-b598-99e9e1d8c012","Type":"ContainerDied","Data":"f8b2f19a9da8a63f4dfbfa0d937fcf82bfc285f33c51061861063e07b987cf7e"} Oct 30 00:39:43.191462 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.191425 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t" event={"ID":"a60854fb-b67c-48f0-b598-99e9e1d8c012","Type":"ContainerDied","Data":"7d9875d11b7f8234c9206ad16e39ffc4cc81c425e24cd95b3b4acdcdce30551c"} Oct 30 00:39:43.191462 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.191444 5001 scope.go:122] "RemoveContainer" containerID="f8b2f19a9da8a63f4dfbfa0d937fcf82bfc285f33c51061861063e07b987cf7e" Oct 30 00:39:43.203595 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.203575 5001 scope.go:122] "RemoveContainer" containerID="f8b2f19a9da8a63f4dfbfa0d937fcf82bfc285f33c51061861063e07b987cf7e" Oct 30 00:39:43.203876 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:39:43.203854 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8b2f19a9da8a63f4dfbfa0d937fcf82bfc285f33c51061861063e07b987cf7e\": container with ID starting with f8b2f19a9da8a63f4dfbfa0d937fcf82bfc285f33c51061861063e07b987cf7e not found: ID does not exist" containerID="f8b2f19a9da8a63f4dfbfa0d937fcf82bfc285f33c51061861063e07b987cf7e" Oct 30 00:39:43.203970 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.203885 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8b2f19a9da8a63f4dfbfa0d937fcf82bfc285f33c51061861063e07b987cf7e"} err="failed to get container status \"f8b2f19a9da8a63f4dfbfa0d937fcf82bfc285f33c51061861063e07b987cf7e\": rpc error: code = NotFound desc = could not find container \"f8b2f19a9da8a63f4dfbfa0d937fcf82bfc285f33c51061861063e07b987cf7e\": container with ID starting with f8b2f19a9da8a63f4dfbfa0d937fcf82bfc285f33c51061861063e07b987cf7e not found: ID does not exist" Oct 30 00:39:43.236286 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.236262 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t"] Oct 30 00:39:43.245636 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.245607 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-d96837/autores-model-d96837-kserve-7d744b4869-ft29t"] Oct 30 00:39:43.303766 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.303731 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-d96837-5565b8fc78-qwjl5" podUID="cd129c0a-51ff-4cd3-85b9-25c05b886998" containerName="payload-processing" containerID="cri-o://2ebd72563cb6c7379586440b60469dd1f022b6d8d78aba16e1b00ed4f9c09af1" gracePeriod=25 Oct 30 00:39:43.323242 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.323212 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-d96837-55b85d6d8c-tgmp9" podUID="0d81642d-32eb-45e9-816b-dee60db16716" containerName="payload-pre-processing" containerID="cri-o://8430c9b9da857b6f4cdfcd100205f7b38b7c6ad6c610d86105c82b9773d03fe2" gracePeriod=25 Oct 30 00:39:43.569399 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.569371 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-d96837-5565b8fc78-qwjl5" Oct 30 00:39:43.628102 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.628076 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-d96837-55b85d6d8c-tgmp9" Oct 30 00:39:43.741548 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.741461 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd129c0a-51ff-4cd3-85b9-25c05b886998-kube-api-access-7sk88\" (UniqueName: \"kubernetes.io/projected/cd129c0a-51ff-4cd3-85b9-25c05b886998-kube-api-access-7sk88\") pod \"cd129c0a-51ff-4cd3-85b9-25c05b886998\" (UID: \"cd129c0a-51ff-4cd3-85b9-25c05b886998\") " Oct 30 00:39:43.741714 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.741593 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d81642d-32eb-45e9-816b-dee60db16716-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d81642d-32eb-45e9-816b-dee60db16716-plugins-config-volume\") pod \"0d81642d-32eb-45e9-816b-dee60db16716\" (UID: \"0d81642d-32eb-45e9-816b-dee60db16716\") " Oct 30 00:39:43.741714 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.741617 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d81642d-32eb-45e9-816b-dee60db16716-kube-api-access-w22gq\" (UniqueName: \"kubernetes.io/projected/0d81642d-32eb-45e9-816b-dee60db16716-kube-api-access-w22gq\") pod \"0d81642d-32eb-45e9-816b-dee60db16716\" (UID: \"0d81642d-32eb-45e9-816b-dee60db16716\") " Oct 30 00:39:43.741714 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.741663 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd129c0a-51ff-4cd3-85b9-25c05b886998-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd129c0a-51ff-4cd3-85b9-25c05b886998-plugins-config-volume\") pod \"cd129c0a-51ff-4cd3-85b9-25c05b886998\" (UID: \"cd129c0a-51ff-4cd3-85b9-25c05b886998\") " Oct 30 00:39:43.742024 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.741996 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d81642d-32eb-45e9-816b-dee60db16716-plugins-config-volume" pod "0d81642d-32eb-45e9-816b-dee60db16716" (UID: "0d81642d-32eb-45e9-816b-dee60db16716"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:43.742096 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.742040 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd129c0a-51ff-4cd3-85b9-25c05b886998-plugins-config-volume" pod "cd129c0a-51ff-4cd3-85b9-25c05b886998" (UID: "cd129c0a-51ff-4cd3-85b9-25c05b886998"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:43.743763 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.743738 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d81642d-32eb-45e9-816b-dee60db16716-kube-api-access-w22gq" pod "0d81642d-32eb-45e9-816b-dee60db16716" (UID: "0d81642d-32eb-45e9-816b-dee60db16716"). InnerVolumeSpecName "kube-api-access-w22gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:43.743892 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.743768 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd129c0a-51ff-4cd3-85b9-25c05b886998-kube-api-access-7sk88" pod "cd129c0a-51ff-4cd3-85b9-25c05b886998" (UID: "cd129c0a-51ff-4cd3-85b9-25c05b886998"). InnerVolumeSpecName "kube-api-access-7sk88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:43.842780 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.842750 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d81642d-32eb-45e9-816b-dee60db16716-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:43.842780 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.842778 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w22gq\" (UniqueName: \"kubernetes.io/projected/0d81642d-32eb-45e9-816b-dee60db16716-kube-api-access-w22gq\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:43.842780 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.842788 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd129c0a-51ff-4cd3-85b9-25c05b886998-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:43.843035 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:43.842798 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sk88\" (UniqueName: \"kubernetes.io/projected/cd129c0a-51ff-4cd3-85b9-25c05b886998-kube-api-access-7sk88\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:44.200485 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:44.200454 5001 generic.go:356] "Generic (PLEG): container finished" podID="0d81642d-32eb-45e9-816b-dee60db16716" containerID="8430c9b9da857b6f4cdfcd100205f7b38b7c6ad6c610d86105c82b9773d03fe2" exitCode=0 Oct 30 00:39:44.200659 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:44.200496 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-d96837-55b85d6d8c-tgmp9" Oct 30 00:39:44.200659 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:44.200541 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-d96837-55b85d6d8c-tgmp9" event={"ID":"0d81642d-32eb-45e9-816b-dee60db16716","Type":"ContainerDied","Data":"8430c9b9da857b6f4cdfcd100205f7b38b7c6ad6c610d86105c82b9773d03fe2"} Oct 30 00:39:44.200659 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:44.200592 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-d96837-55b85d6d8c-tgmp9" event={"ID":"0d81642d-32eb-45e9-816b-dee60db16716","Type":"ContainerDied","Data":"d312a2cdef32a8840474ea2789439a6ee2fa63c664763280de0b465524a90f96"} Oct 30 00:39:44.200659 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:44.200621 5001 scope.go:122] "RemoveContainer" containerID="8430c9b9da857b6f4cdfcd100205f7b38b7c6ad6c610d86105c82b9773d03fe2" Oct 30 00:39:44.202398 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:44.202375 5001 generic.go:356] "Generic (PLEG): container finished" podID="cd129c0a-51ff-4cd3-85b9-25c05b886998" containerID="2ebd72563cb6c7379586440b60469dd1f022b6d8d78aba16e1b00ed4f9c09af1" exitCode=0 Oct 30 00:39:44.202505 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:44.202416 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-d96837-5565b8fc78-qwjl5" Oct 30 00:39:44.202505 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:44.202461 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-d96837-5565b8fc78-qwjl5" event={"ID":"cd129c0a-51ff-4cd3-85b9-25c05b886998","Type":"ContainerDied","Data":"2ebd72563cb6c7379586440b60469dd1f022b6d8d78aba16e1b00ed4f9c09af1"} Oct 30 00:39:44.202656 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:44.202515 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-d96837-5565b8fc78-qwjl5" event={"ID":"cd129c0a-51ff-4cd3-85b9-25c05b886998","Type":"ContainerDied","Data":"afaa5db926293590930bc44908d66375f070451362a032e95c48fd661137551f"} Oct 30 00:39:44.212652 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:44.212622 5001 scope.go:122] "RemoveContainer" containerID="8430c9b9da857b6f4cdfcd100205f7b38b7c6ad6c610d86105c82b9773d03fe2" Oct 30 00:39:44.212977 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:39:44.212950 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8430c9b9da857b6f4cdfcd100205f7b38b7c6ad6c610d86105c82b9773d03fe2\": container with ID starting with 8430c9b9da857b6f4cdfcd100205f7b38b7c6ad6c610d86105c82b9773d03fe2 not found: ID does not exist" containerID="8430c9b9da857b6f4cdfcd100205f7b38b7c6ad6c610d86105c82b9773d03fe2" Oct 30 00:39:44.213067 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:44.212985 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8430c9b9da857b6f4cdfcd100205f7b38b7c6ad6c610d86105c82b9773d03fe2"} err="failed to get container status \"8430c9b9da857b6f4cdfcd100205f7b38b7c6ad6c610d86105c82b9773d03fe2\": rpc error: code = NotFound desc = could not find container \"8430c9b9da857b6f4cdfcd100205f7b38b7c6ad6c610d86105c82b9773d03fe2\": container with ID starting with 8430c9b9da857b6f4cdfcd100205f7b38b7c6ad6c610d86105c82b9773d03fe2 not found: ID does not exist" Oct 30 00:39:44.213067 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:44.213008 5001 scope.go:122] "RemoveContainer" containerID="2ebd72563cb6c7379586440b60469dd1f022b6d8d78aba16e1b00ed4f9c09af1" Oct 30 00:39:44.222788 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:44.222769 5001 scope.go:122] "RemoveContainer" containerID="2ebd72563cb6c7379586440b60469dd1f022b6d8d78aba16e1b00ed4f9c09af1" Oct 30 00:39:44.223094 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:39:44.223075 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ebd72563cb6c7379586440b60469dd1f022b6d8d78aba16e1b00ed4f9c09af1\": container with ID starting with 2ebd72563cb6c7379586440b60469dd1f022b6d8d78aba16e1b00ed4f9c09af1 not found: ID does not exist" containerID="2ebd72563cb6c7379586440b60469dd1f022b6d8d78aba16e1b00ed4f9c09af1" Oct 30 00:39:44.223162 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:44.223101 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ebd72563cb6c7379586440b60469dd1f022b6d8d78aba16e1b00ed4f9c09af1"} err="failed to get container status \"2ebd72563cb6c7379586440b60469dd1f022b6d8d78aba16e1b00ed4f9c09af1\": rpc error: code = NotFound desc = could not find container \"2ebd72563cb6c7379586440b60469dd1f022b6d8d78aba16e1b00ed4f9c09af1\": container with ID starting with 2ebd72563cb6c7379586440b60469dd1f022b6d8d78aba16e1b00ed4f9c09af1 not found: ID does not exist" Oct 30 00:39:44.276674 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:44.276649 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d96837-55b85d6d8c-tgmp9"] Oct 30 00:39:44.335755 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:44.335731 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d96837-55b85d6d8c-tgmp9"] Oct 30 00:39:44.364842 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:44.364811 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-d96837-5565b8fc78-qwjl5"] Oct 30 00:39:44.370212 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:44.370186 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-d96837-5565b8fc78-qwjl5"] Oct 30 00:39:44.929189 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:44.929150 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d81642d-32eb-45e9-816b-dee60db16716" path="/var/lib/kubelet/pods/0d81642d-32eb-45e9-816b-dee60db16716/volumes" Oct 30 00:39:44.929738 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:44.929719 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a60854fb-b67c-48f0-b598-99e9e1d8c012" path="/var/lib/kubelet/pods/a60854fb-b67c-48f0-b598-99e9e1d8c012/volumes" Oct 30 00:39:44.930219 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:44.930203 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd129c0a-51ff-4cd3-85b9-25c05b886998" path="/var/lib/kubelet/pods/cd129c0a-51ff-4cd3-85b9-25c05b886998/volumes" Oct 30 00:39:45.448558 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:45.448533 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8gb5_9fa688d8-29ca-4c1f-926a-f1a4099e6b44/console-operator/2.log" Oct 30 00:39:45.458724 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:45.458701 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8gb5_9fa688d8-29ca-4c1f-926a-f1a4099e6b44/console-operator/2.log" Oct 30 00:39:46.876827 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:46.876790 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h"] Oct 30 00:39:46.877353 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:46.877107 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" podUID="6cad4256-1e03-4119-9cbe-4dbe1717600a" containerName="istio-proxy" containerID="cri-o://5a97e46cc2da747ae8e2780e7af5ca6b57c28a9782bc24b4cd0346e30089a552" gracePeriod=30 Oct 30 00:39:47.322508 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:39:47.322475 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:39:47.501624 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.501590 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-63e288-9dznp"] Oct 30 00:39:47.502033 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.502020 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4c415517-5484-4905-b9df-8510d1da9ddb" containerName="istio-proxy" Oct 30 00:39:47.502089 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.502034 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c415517-5484-4905-b9df-8510d1da9ddb" containerName="istio-proxy" Oct 30 00:39:47.502089 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.502055 5001 cpu_manager.go:414] "containerMap: removing container" podUID="0d81642d-32eb-45e9-816b-dee60db16716" containerName="payload-pre-processing" Oct 30 00:39:47.502089 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.502060 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d81642d-32eb-45e9-816b-dee60db16716" containerName="payload-pre-processing" Oct 30 00:39:47.502089 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.502074 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f94ceedc-64a2-4a09-816f-42ad1b823a34" containerName="revoke-keys" Oct 30 00:39:47.502089 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.502079 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f94ceedc-64a2-4a09-816f-42ad1b823a34" containerName="revoke-keys" Oct 30 00:39:47.502270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.502097 5001 cpu_manager.go:414] "containerMap: removing container" podUID="a60854fb-b67c-48f0-b598-99e9e1d8c012" containerName="main" Oct 30 00:39:47.502270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.502102 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a60854fb-b67c-48f0-b598-99e9e1d8c012" containerName="main" Oct 30 00:39:47.502270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.502109 5001 cpu_manager.go:414] "containerMap: removing container" podUID="27699192-a6f9-4c03-a63d-e19a122dff3a" containerName="istio-proxy" Oct 30 00:39:47.502270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.502114 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27699192-a6f9-4c03-a63d-e19a122dff3a" containerName="istio-proxy" Oct 30 00:39:47.502270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.502122 5001 cpu_manager.go:414] "containerMap: removing container" podUID="cd129c0a-51ff-4cd3-85b9-25c05b886998" containerName="payload-processing" Oct 30 00:39:47.502270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.502126 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd129c0a-51ff-4cd3-85b9-25c05b886998" containerName="payload-processing" Oct 30 00:39:47.502270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.502134 5001 cpu_manager.go:414] "containerMap: removing container" podUID="9306cc6a-b126-4e05-b726-fc9b267a0d78" containerName="payload-pre-processing" Oct 30 00:39:47.502270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.502138 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9306cc6a-b126-4e05-b726-fc9b267a0d78" containerName="payload-pre-processing" Oct 30 00:39:47.502270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.502146 5001 cpu_manager.go:414] "containerMap: removing container" podUID="91e5f27d-f718-4239-98ee-b0a17382677e" containerName="payload-processing" Oct 30 00:39:47.502270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.502150 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91e5f27d-f718-4239-98ee-b0a17382677e" containerName="payload-processing" Oct 30 00:39:47.502270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.502157 5001 cpu_manager.go:414] "containerMap: removing container" podUID="339eea71-a503-460d-b45e-94561c780aca" containerName="payload-pre-processing" Oct 30 00:39:47.502270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.502161 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="339eea71-a503-460d-b45e-94561c780aca" containerName="payload-pre-processing" Oct 30 00:39:47.502270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.502227 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a60854fb-b67c-48f0-b598-99e9e1d8c012" containerName="main" Oct 30 00:39:47.502270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.502234 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="27699192-a6f9-4c03-a63d-e19a122dff3a" containerName="istio-proxy" Oct 30 00:39:47.502270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.502241 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd129c0a-51ff-4cd3-85b9-25c05b886998" containerName="payload-processing" Oct 30 00:39:47.502270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.502246 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9306cc6a-b126-4e05-b726-fc9b267a0d78" containerName="payload-pre-processing" Oct 30 00:39:47.502270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.502254 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="91e5f27d-f718-4239-98ee-b0a17382677e" containerName="payload-processing" Oct 30 00:39:47.502270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.502261 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="339eea71-a503-460d-b45e-94561c780aca" containerName="payload-pre-processing" Oct 30 00:39:47.502270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.502269 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c415517-5484-4905-b9df-8510d1da9ddb" containerName="istio-proxy" Oct 30 00:39:47.502270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.502276 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d81642d-32eb-45e9-816b-dee60db16716" containerName="payload-pre-processing" Oct 30 00:39:47.503049 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.502284 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f94ceedc-64a2-4a09-816f-42ad1b823a34" containerName="revoke-keys" Oct 30 00:39:47.506998 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.506981 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-63e288-9dznp" Oct 30 00:39:47.519306 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.519282 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:39:47.530965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.530940 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:39:47.531142 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.531126 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:39:47.533642 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.533618 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-63e288-9dznp"] Oct 30 00:39:47.619554 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.619477 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:39:47.681185 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.681150 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d84398a8-8d2b-4047-829b-ef13ddf14a35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-63e288-9dznp\" (UID: \"d84398a8-8d2b-4047-829b-ef13ddf14a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-63e288-9dznp" Oct 30 00:39:47.681375 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.681293 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffpzh\" (UniqueName: \"kubernetes.io/projected/d84398a8-8d2b-4047-829b-ef13ddf14a35-kube-api-access-ffpzh\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-63e288-9dznp\" (UID: \"d84398a8-8d2b-4047-829b-ef13ddf14a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-63e288-9dznp" Oct 30 00:39:47.681375 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.681339 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d84398a8-8d2b-4047-829b-ef13ddf14a35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-63e288-9dznp\" (UID: \"d84398a8-8d2b-4047-829b-ef13ddf14a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-63e288-9dznp" Oct 30 00:39:47.782020 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.781984 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d84398a8-8d2b-4047-829b-ef13ddf14a35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-63e288-9dznp\" (UID: \"d84398a8-8d2b-4047-829b-ef13ddf14a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-63e288-9dznp" Oct 30 00:39:47.782229 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.782053 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffpzh\" (UniqueName: \"kubernetes.io/projected/d84398a8-8d2b-4047-829b-ef13ddf14a35-kube-api-access-ffpzh\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-63e288-9dznp\" (UID: \"d84398a8-8d2b-4047-829b-ef13ddf14a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-63e288-9dznp" Oct 30 00:39:47.782229 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.782076 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d84398a8-8d2b-4047-829b-ef13ddf14a35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-63e288-9dznp\" (UID: \"d84398a8-8d2b-4047-829b-ef13ddf14a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-63e288-9dznp" Oct 30 00:39:47.782822 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.782801 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d84398a8-8d2b-4047-829b-ef13ddf14a35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-63e288-9dznp\" (UID: \"d84398a8-8d2b-4047-829b-ef13ddf14a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-63e288-9dznp" Oct 30 00:39:47.784403 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.784379 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d84398a8-8d2b-4047-829b-ef13ddf14a35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-63e288-9dznp\" (UID: \"d84398a8-8d2b-4047-829b-ef13ddf14a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-63e288-9dznp" Oct 30 00:39:47.805187 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.805156 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffpzh\" (UniqueName: \"kubernetes.io/projected/d84398a8-8d2b-4047-829b-ef13ddf14a35-kube-api-access-ffpzh\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-63e288-9dznp\" (UID: \"d84398a8-8d2b-4047-829b-ef13ddf14a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-63e288-9dznp" Oct 30 00:39:47.821102 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:47.821079 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-63e288-9dznp" Oct 30 00:39:48.120204 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:48.120156 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-63e288-9dznp"] Oct 30 00:39:48.120560 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:39:48.120537 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd84398a8_8d2b_4047_829b_ef13ddf14a35.slice/crio-7693ff8902c276ca04ff82c16a544e313ce90489dd68496ec93890226e9f86a1 WatchSource:0}: Error finding container 7693ff8902c276ca04ff82c16a544e313ce90489dd68496ec93890226e9f86a1: Status 404 returned error can't find the container with id 7693ff8902c276ca04ff82c16a544e313ce90489dd68496ec93890226e9f86a1 Oct 30 00:39:48.237303 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:48.237270 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-63e288-9dznp" event={"ID":"d84398a8-8d2b-4047-829b-ef13ddf14a35","Type":"ContainerStarted","Data":"7693ff8902c276ca04ff82c16a544e313ce90489dd68496ec93890226e9f86a1"} Oct 30 00:39:49.246645 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:49.246555 5001 generic.go:356] "Generic (PLEG): container finished" podID="d84398a8-8d2b-4047-829b-ef13ddf14a35" containerID="61c90446b2fd92e43bb9602f695408b04f998922f924277cde6f7fa533efb761" exitCode=0 Oct 30 00:39:49.246645 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:49.246608 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-63e288-9dznp" event={"ID":"d84398a8-8d2b-4047-829b-ef13ddf14a35","Type":"ContainerDied","Data":"61c90446b2fd92e43bb9602f695408b04f998922f924277cde6f7fa533efb761"} Oct 30 00:39:50.399596 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:50.399568 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-63e288-9dznp" Oct 30 00:39:50.510403 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:50.510380 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d84398a8-8d2b-4047-829b-ef13ddf14a35-kube-api-access-ffpzh\" (UniqueName: \"kubernetes.io/projected/d84398a8-8d2b-4047-829b-ef13ddf14a35-kube-api-access-ffpzh\") pod \"d84398a8-8d2b-4047-829b-ef13ddf14a35\" (UID: \"d84398a8-8d2b-4047-829b-ef13ddf14a35\") " Oct 30 00:39:50.510575 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:50.510438 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d84398a8-8d2b-4047-829b-ef13ddf14a35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d84398a8-8d2b-4047-829b-ef13ddf14a35-maas-api-xks-ca\") pod \"d84398a8-8d2b-4047-829b-ef13ddf14a35\" (UID: \"d84398a8-8d2b-4047-829b-ef13ddf14a35\") " Oct 30 00:39:50.510575 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:50.510540 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d84398a8-8d2b-4047-829b-ef13ddf14a35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d84398a8-8d2b-4047-829b-ef13ddf14a35-maas-api-openshift-ca\") pod \"d84398a8-8d2b-4047-829b-ef13ddf14a35\" (UID: \"d84398a8-8d2b-4047-829b-ef13ddf14a35\") " Oct 30 00:39:50.510976 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:50.510948 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d84398a8-8d2b-4047-829b-ef13ddf14a35-maas-api-openshift-ca" pod "d84398a8-8d2b-4047-829b-ef13ddf14a35" (UID: "d84398a8-8d2b-4047-829b-ef13ddf14a35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:50.512501 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:50.512476 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d84398a8-8d2b-4047-829b-ef13ddf14a35-maas-api-xks-ca" pod "d84398a8-8d2b-4047-829b-ef13ddf14a35" (UID: "d84398a8-8d2b-4047-829b-ef13ddf14a35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:39:50.512580 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:50.512507 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d84398a8-8d2b-4047-829b-ef13ddf14a35-kube-api-access-ffpzh" pod "d84398a8-8d2b-4047-829b-ef13ddf14a35" (UID: "d84398a8-8d2b-4047-829b-ef13ddf14a35"). InnerVolumeSpecName "kube-api-access-ffpzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:50.612404 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:50.612373 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d84398a8-8d2b-4047-829b-ef13ddf14a35-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:50.612404 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:50.612400 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffpzh\" (UniqueName: \"kubernetes.io/projected/d84398a8-8d2b-4047-829b-ef13ddf14a35-kube-api-access-ffpzh\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:50.612404 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:50.612410 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d84398a8-8d2b-4047-829b-ef13ddf14a35-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:51.263590 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:51.263561 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-63e288-9dznp" Oct 30 00:39:51.263775 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:51.263587 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-63e288-9dznp" event={"ID":"d84398a8-8d2b-4047-829b-ef13ddf14a35","Type":"ContainerDied","Data":"7693ff8902c276ca04ff82c16a544e313ce90489dd68496ec93890226e9f86a1"} Oct 30 00:39:51.263775 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:51.263638 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7693ff8902c276ca04ff82c16a544e313ce90489dd68496ec93890226e9f86a1" Oct 30 00:39:52.139084 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.139056 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:39:52.273069 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.273037 5001 generic.go:356] "Generic (PLEG): container finished" podID="6cad4256-1e03-4119-9cbe-4dbe1717600a" containerID="5a97e46cc2da747ae8e2780e7af5ca6b57c28a9782bc24b4cd0346e30089a552" exitCode=0 Oct 30 00:39:52.273242 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.273091 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" Oct 30 00:39:52.273242 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.273117 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" event={"ID":"6cad4256-1e03-4119-9cbe-4dbe1717600a","Type":"ContainerDied","Data":"5a97e46cc2da747ae8e2780e7af5ca6b57c28a9782bc24b4cd0346e30089a552"} Oct 30 00:39:52.273242 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.273171 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h" event={"ID":"6cad4256-1e03-4119-9cbe-4dbe1717600a","Type":"ContainerDied","Data":"957b4af3da09d5e5dd01792f0ae4f7eed861623b1286152a755975c01b48a8c9"} Oct 30 00:39:52.273242 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.273194 5001 scope.go:122] "RemoveContainer" containerID="5a97e46cc2da747ae8e2780e7af5ca6b57c28a9782bc24b4cd0346e30089a552" Oct 30 00:39:52.283776 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.283758 5001 scope.go:122] "RemoveContainer" containerID="5a1ecd4d86d27be3c301d8922fa4238fdd9622d99a7e18606a26bf98e429130b" Oct 30 00:39:52.294053 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.294013 5001 scope.go:122] "RemoveContainer" containerID="5a97e46cc2da747ae8e2780e7af5ca6b57c28a9782bc24b4cd0346e30089a552" Oct 30 00:39:52.294344 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:39:52.294324 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a97e46cc2da747ae8e2780e7af5ca6b57c28a9782bc24b4cd0346e30089a552\": container with ID starting with 5a97e46cc2da747ae8e2780e7af5ca6b57c28a9782bc24b4cd0346e30089a552 not found: ID does not exist" containerID="5a97e46cc2da747ae8e2780e7af5ca6b57c28a9782bc24b4cd0346e30089a552" Oct 30 00:39:52.294386 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.294354 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a97e46cc2da747ae8e2780e7af5ca6b57c28a9782bc24b4cd0346e30089a552"} err="failed to get container status \"5a97e46cc2da747ae8e2780e7af5ca6b57c28a9782bc24b4cd0346e30089a552\": rpc error: code = NotFound desc = could not find container \"5a97e46cc2da747ae8e2780e7af5ca6b57c28a9782bc24b4cd0346e30089a552\": container with ID starting with 5a97e46cc2da747ae8e2780e7af5ca6b57c28a9782bc24b4cd0346e30089a552 not found: ID does not exist" Oct 30 00:39:52.294386 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.294372 5001 scope.go:122] "RemoveContainer" containerID="5a1ecd4d86d27be3c301d8922fa4238fdd9622d99a7e18606a26bf98e429130b" Oct 30 00:39:52.294590 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:39:52.294576 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a1ecd4d86d27be3c301d8922fa4238fdd9622d99a7e18606a26bf98e429130b\": container with ID starting with 5a1ecd4d86d27be3c301d8922fa4238fdd9622d99a7e18606a26bf98e429130b not found: ID does not exist" containerID="5a1ecd4d86d27be3c301d8922fa4238fdd9622d99a7e18606a26bf98e429130b" Oct 30 00:39:52.294629 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.294593 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a1ecd4d86d27be3c301d8922fa4238fdd9622d99a7e18606a26bf98e429130b"} err="failed to get container status \"5a1ecd4d86d27be3c301d8922fa4238fdd9622d99a7e18606a26bf98e429130b\": rpc error: code = NotFound desc = could not find container \"5a1ecd4d86d27be3c301d8922fa4238fdd9622d99a7e18606a26bf98e429130b\": container with ID starting with 5a1ecd4d86d27be3c301d8922fa4238fdd9622d99a7e18606a26bf98e429130b not found: ID does not exist" Oct 30 00:39:52.331534 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.331495 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cad4256-1e03-4119-9cbe-4dbe1717600a-kube-api-access-8v2hj\" (UniqueName: \"kubernetes.io/projected/6cad4256-1e03-4119-9cbe-4dbe1717600a-kube-api-access-8v2hj\") pod \"6cad4256-1e03-4119-9cbe-4dbe1717600a\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " Oct 30 00:39:52.331703 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.331568 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-credential-socket\") pod \"6cad4256-1e03-4119-9cbe-4dbe1717600a\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " Oct 30 00:39:52.331703 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.331607 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-istio-envoy\") pod \"6cad4256-1e03-4119-9cbe-4dbe1717600a\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " Oct 30 00:39:52.331703 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.331639 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-istio-data\") pod \"6cad4256-1e03-4119-9cbe-4dbe1717600a\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " Oct 30 00:39:52.331898 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.331770 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-workload-socket\") pod \"6cad4256-1e03-4119-9cbe-4dbe1717600a\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " Oct 30 00:39:52.331898 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.331823 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6cad4256-1e03-4119-9cbe-4dbe1717600a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cad4256-1e03-4119-9cbe-4dbe1717600a-istio-podinfo\") pod \"6cad4256-1e03-4119-9cbe-4dbe1717600a\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " Oct 30 00:39:52.331898 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.331872 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cad4256-1e03-4119-9cbe-4dbe1717600a-istio-token\" (UniqueName: \"kubernetes.io/projected/6cad4256-1e03-4119-9cbe-4dbe1717600a-istio-token\") pod \"6cad4256-1e03-4119-9cbe-4dbe1717600a\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " Oct 30 00:39:52.331898 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.331877 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-credential-socket" pod "6cad4256-1e03-4119-9cbe-4dbe1717600a" (UID: "6cad4256-1e03-4119-9cbe-4dbe1717600a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:52.332158 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.331939 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-workload-certs\") pod \"6cad4256-1e03-4119-9cbe-4dbe1717600a\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " Oct 30 00:39:52.332158 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.331993 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cad4256-1e03-4119-9cbe-4dbe1717600a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cad4256-1e03-4119-9cbe-4dbe1717600a-istiod-ca-cert\") pod \"6cad4256-1e03-4119-9cbe-4dbe1717600a\" (UID: \"6cad4256-1e03-4119-9cbe-4dbe1717600a\") " Oct 30 00:39:52.332457 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.332411 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:52.332588 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.332504 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cad4256-1e03-4119-9cbe-4dbe1717600a-istiod-ca-cert" pod "6cad4256-1e03-4119-9cbe-4dbe1717600a" (UID: "6cad4256-1e03-4119-9cbe-4dbe1717600a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:52.332588 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.332513 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-istio-data" pod "6cad4256-1e03-4119-9cbe-4dbe1717600a" (UID: "6cad4256-1e03-4119-9cbe-4dbe1717600a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:52.332588 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.332569 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-workload-certs" pod "6cad4256-1e03-4119-9cbe-4dbe1717600a" (UID: "6cad4256-1e03-4119-9cbe-4dbe1717600a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:52.332588 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.332582 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-workload-socket" pod "6cad4256-1e03-4119-9cbe-4dbe1717600a" (UID: "6cad4256-1e03-4119-9cbe-4dbe1717600a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:52.334429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.334381 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cad4256-1e03-4119-9cbe-4dbe1717600a-kube-api-access-8v2hj" pod "6cad4256-1e03-4119-9cbe-4dbe1717600a" (UID: "6cad4256-1e03-4119-9cbe-4dbe1717600a"). InnerVolumeSpecName "kube-api-access-8v2hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:52.334524 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.334505 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-istio-envoy" pod "6cad4256-1e03-4119-9cbe-4dbe1717600a" (UID: "6cad4256-1e03-4119-9cbe-4dbe1717600a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:52.334880 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.334860 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cad4256-1e03-4119-9cbe-4dbe1717600a-istio-token" pod "6cad4256-1e03-4119-9cbe-4dbe1717600a" (UID: "6cad4256-1e03-4119-9cbe-4dbe1717600a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:52.334993 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.334883 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6cad4256-1e03-4119-9cbe-4dbe1717600a-istio-podinfo" pod "6cad4256-1e03-4119-9cbe-4dbe1717600a" (UID: "6cad4256-1e03-4119-9cbe-4dbe1717600a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:39:52.433250 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.433212 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v2hj\" (UniqueName: \"kubernetes.io/projected/6cad4256-1e03-4119-9cbe-4dbe1717600a-kube-api-access-8v2hj\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:52.433250 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.433245 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:52.433250 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.433255 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:52.433507 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.433263 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:52.433507 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.433273 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cad4256-1e03-4119-9cbe-4dbe1717600a-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:52.433507 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.433282 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cad4256-1e03-4119-9cbe-4dbe1717600a-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:52.433507 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.433290 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cad4256-1e03-4119-9cbe-4dbe1717600a-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:52.433507 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.433298 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cad4256-1e03-4119-9cbe-4dbe1717600a-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:39:52.617420 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.617385 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h"] Oct 30 00:39:52.625241 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.625212 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-d96837-openshift-default-7978c98dcc-w445h"] Oct 30 00:39:52.917763 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:52.917685 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cad4256-1e03-4119-9cbe-4dbe1717600a" path="/var/lib/kubelet/pods/6cad4256-1e03-4119-9cbe-4dbe1717600a/volumes" Oct 30 00:39:59.267344 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:59.267259 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-63e288-9dznp"] Oct 30 00:39:59.276436 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:39:59.276406 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-63e288-9dznp"] Oct 30 00:40:00.917339 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:00.917299 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d84398a8-8d2b-4047-829b-ef13ddf14a35" path="/var/lib/kubelet/pods/d84398a8-8d2b-4047-829b-ef13ddf14a35/volumes" Oct 30 00:40:01.724600 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:01.724569 5001 scope.go:122] "RemoveContainer" containerID="845d25c8a2445cf8c863fde6a4eee6d96a356c356e4ef6c2b93321f82614c778" Oct 30 00:40:01.735074 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:01.735054 5001 scope.go:122] "RemoveContainer" containerID="92be57859ada05e4f289424fb9d02a42a08f5710838ec08a7a374739664c9d3a" Oct 30 00:40:01.744514 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:01.744495 5001 scope.go:122] "RemoveContainer" containerID="b4e288c518f6f4080239560e51975cd36b65a01d961d3d936ad83e00a9ef5b0f" Oct 30 00:40:01.753464 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:01.753446 5001 scope.go:122] "RemoveContainer" containerID="56b2bfcf34ff59af50544c34c8e0f8981dc208cbb20cf97436220bcc373df1b1" Oct 30 00:40:01.762473 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:01.762454 5001 scope.go:122] "RemoveContainer" containerID="8f8bf97b9dbfe60c70fbc1d12b2ed08bf008960ac45c2ef62a0f7455fc10a76e" Oct 30 00:40:01.770970 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:01.770953 5001 scope.go:122] "RemoveContainer" containerID="8e1217c5d6d8f8035905748e4834e9f91f224196f01c80ffa524a945a026ad68" Oct 30 00:40:01.779997 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:01.779977 5001 scope.go:122] "RemoveContainer" containerID="7122efbb1cf2e6f43e62091d7b67135731b5975c9b534f5a43cd0d76fa6cabb6" Oct 30 00:40:06.050620 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:06.050586 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p"] Oct 30 00:40:06.051403 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:06.051370 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" podUID="aadaf6fb-2702-4c04-88c9-cbb181049d68" containerName="istio-proxy" containerID="cri-o://eeaebb279140456c722be18222cfe34fe4c8f4092ef689e75a754e377382c219" gracePeriod=30 Oct 30 00:40:06.658805 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:06.658767 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f1268c-rhx28"] Oct 30 00:40:06.659241 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:06.659226 5001 cpu_manager.go:414] "containerMap: removing container" podUID="6cad4256-1e03-4119-9cbe-4dbe1717600a" containerName="istio-proxy" Oct 30 00:40:06.659307 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:06.659242 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cad4256-1e03-4119-9cbe-4dbe1717600a" containerName="istio-proxy" Oct 30 00:40:06.659307 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:06.659267 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d84398a8-8d2b-4047-829b-ef13ddf14a35" containerName="revoke-keys" Oct 30 00:40:06.659307 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:06.659272 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d84398a8-8d2b-4047-829b-ef13ddf14a35" containerName="revoke-keys" Oct 30 00:40:06.659407 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:06.659346 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cad4256-1e03-4119-9cbe-4dbe1717600a" containerName="istio-proxy" Oct 30 00:40:06.659407 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:06.659361 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d84398a8-8d2b-4047-829b-ef13ddf14a35" containerName="revoke-keys" Oct 30 00:40:06.666215 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:06.666194 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f1268c-rhx28" Oct 30 00:40:06.680349 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:06.680330 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:40:06.681092 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:06.681077 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:40:06.695854 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:06.695825 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f1268c-rhx28"] Oct 30 00:40:06.724885 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:06.724861 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:40:06.725034 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:06.724993 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:40:06.782594 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:06.782563 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53e4314e-f84a-4f80-8f61-d847f6425ff8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f1268c-rhx28\" (UID: \"53e4314e-f84a-4f80-8f61-d847f6425ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f1268c-rhx28" Oct 30 00:40:06.782739 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:06.782677 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53e4314e-f84a-4f80-8f61-d847f6425ff8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f1268c-rhx28\" (UID: \"53e4314e-f84a-4f80-8f61-d847f6425ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f1268c-rhx28" Oct 30 00:40:06.782739 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:06.782699 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq7zh\" (UniqueName: \"kubernetes.io/projected/53e4314e-f84a-4f80-8f61-d847f6425ff8-kube-api-access-dq7zh\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f1268c-rhx28\" (UID: \"53e4314e-f84a-4f80-8f61-d847f6425ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f1268c-rhx28" Oct 30 00:40:06.884180 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:06.884137 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53e4314e-f84a-4f80-8f61-d847f6425ff8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f1268c-rhx28\" (UID: \"53e4314e-f84a-4f80-8f61-d847f6425ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f1268c-rhx28" Oct 30 00:40:06.884380 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:06.884189 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq7zh\" (UniqueName: \"kubernetes.io/projected/53e4314e-f84a-4f80-8f61-d847f6425ff8-kube-api-access-dq7zh\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f1268c-rhx28\" (UID: \"53e4314e-f84a-4f80-8f61-d847f6425ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f1268c-rhx28" Oct 30 00:40:06.884380 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:06.884278 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53e4314e-f84a-4f80-8f61-d847f6425ff8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f1268c-rhx28\" (UID: \"53e4314e-f84a-4f80-8f61-d847f6425ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f1268c-rhx28" Oct 30 00:40:06.884887 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:06.884867 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53e4314e-f84a-4f80-8f61-d847f6425ff8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f1268c-rhx28\" (UID: \"53e4314e-f84a-4f80-8f61-d847f6425ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f1268c-rhx28" Oct 30 00:40:06.886498 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:06.886471 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53e4314e-f84a-4f80-8f61-d847f6425ff8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f1268c-rhx28\" (UID: \"53e4314e-f84a-4f80-8f61-d847f6425ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f1268c-rhx28" Oct 30 00:40:06.904885 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:06.904855 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq7zh\" (UniqueName: \"kubernetes.io/projected/53e4314e-f84a-4f80-8f61-d847f6425ff8-kube-api-access-dq7zh\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f1268c-rhx28\" (UID: \"53e4314e-f84a-4f80-8f61-d847f6425ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f1268c-rhx28" Oct 30 00:40:06.979295 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:06.979232 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f1268c-rhx28" Oct 30 00:40:07.277674 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:07.277644 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f1268c-rhx28"] Oct 30 00:40:07.278888 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:40:07.278865 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53e4314e_f84a_4f80_8f61_d847f6425ff8.slice/crio-dbfb414a9eec16296f74b46fd8db567e3ea8d2a60038975769258c61d8cb6d64 WatchSource:0}: Error finding container dbfb414a9eec16296f74b46fd8db567e3ea8d2a60038975769258c61d8cb6d64: Status 404 returned error can't find the container with id dbfb414a9eec16296f74b46fd8db567e3ea8d2a60038975769258c61d8cb6d64 Oct 30 00:40:07.388333 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:07.388296 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f1268c-rhx28" event={"ID":"53e4314e-f84a-4f80-8f61-d847f6425ff8","Type":"ContainerStarted","Data":"dbfb414a9eec16296f74b46fd8db567e3ea8d2a60038975769258c61d8cb6d64"} Oct 30 00:40:08.396403 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:08.396322 5001 generic.go:356] "Generic (PLEG): container finished" podID="53e4314e-f84a-4f80-8f61-d847f6425ff8" containerID="2bf837513321c829559e12b86be0484f186379b664e5fbb4f54189062cf6b06b" exitCode=0 Oct 30 00:40:08.396403 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:08.396370 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f1268c-rhx28" event={"ID":"53e4314e-f84a-4f80-8f61-d847f6425ff8","Type":"ContainerDied","Data":"2bf837513321c829559e12b86be0484f186379b664e5fbb4f54189062cf6b06b"} Oct 30 00:40:09.542278 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:09.542253 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f1268c-rhx28" Oct 30 00:40:09.613786 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:09.613757 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53e4314e-f84a-4f80-8f61-d847f6425ff8-kube-api-access-dq7zh\" (UniqueName: \"kubernetes.io/projected/53e4314e-f84a-4f80-8f61-d847f6425ff8-kube-api-access-dq7zh\") pod \"53e4314e-f84a-4f80-8f61-d847f6425ff8\" (UID: \"53e4314e-f84a-4f80-8f61-d847f6425ff8\") " Oct 30 00:40:09.613974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:09.613815 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53e4314e-f84a-4f80-8f61-d847f6425ff8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53e4314e-f84a-4f80-8f61-d847f6425ff8-maas-api-xks-ca\") pod \"53e4314e-f84a-4f80-8f61-d847f6425ff8\" (UID: \"53e4314e-f84a-4f80-8f61-d847f6425ff8\") " Oct 30 00:40:09.613974 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:09.613836 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53e4314e-f84a-4f80-8f61-d847f6425ff8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53e4314e-f84a-4f80-8f61-d847f6425ff8-maas-api-openshift-ca\") pod \"53e4314e-f84a-4f80-8f61-d847f6425ff8\" (UID: \"53e4314e-f84a-4f80-8f61-d847f6425ff8\") " Oct 30 00:40:09.614270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:09.614233 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53e4314e-f84a-4f80-8f61-d847f6425ff8-maas-api-openshift-ca" pod "53e4314e-f84a-4f80-8f61-d847f6425ff8" (UID: "53e4314e-f84a-4f80-8f61-d847f6425ff8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:40:09.615953 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:09.615932 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53e4314e-f84a-4f80-8f61-d847f6425ff8-maas-api-xks-ca" pod "53e4314e-f84a-4f80-8f61-d847f6425ff8" (UID: "53e4314e-f84a-4f80-8f61-d847f6425ff8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:40:09.616068 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:09.616008 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53e4314e-f84a-4f80-8f61-d847f6425ff8-kube-api-access-dq7zh" pod "53e4314e-f84a-4f80-8f61-d847f6425ff8" (UID: "53e4314e-f84a-4f80-8f61-d847f6425ff8"). InnerVolumeSpecName "kube-api-access-dq7zh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:40:09.715683 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:09.715613 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq7zh\" (UniqueName: \"kubernetes.io/projected/53e4314e-f84a-4f80-8f61-d847f6425ff8-kube-api-access-dq7zh\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:40:09.715683 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:09.715640 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53e4314e-f84a-4f80-8f61-d847f6425ff8-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:40:09.715683 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:09.715650 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53e4314e-f84a-4f80-8f61-d847f6425ff8-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:40:10.411757 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:10.411726 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f1268c-rhx28" Oct 30 00:40:10.411989 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:10.411767 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f1268c-rhx28" event={"ID":"53e4314e-f84a-4f80-8f61-d847f6425ff8","Type":"ContainerDied","Data":"dbfb414a9eec16296f74b46fd8db567e3ea8d2a60038975769258c61d8cb6d64"} Oct 30 00:40:10.411989 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:10.411816 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbfb414a9eec16296f74b46fd8db567e3ea8d2a60038975769258c61d8cb6d64" Oct 30 00:40:11.302872 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.302849 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:40:11.421107 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.421018 5001 generic.go:356] "Generic (PLEG): container finished" podID="aadaf6fb-2702-4c04-88c9-cbb181049d68" containerID="eeaebb279140456c722be18222cfe34fe4c8f4092ef689e75a754e377382c219" exitCode=0 Oct 30 00:40:11.421107 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.421087 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" Oct 30 00:40:11.421296 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.421100 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" event={"ID":"aadaf6fb-2702-4c04-88c9-cbb181049d68","Type":"ContainerDied","Data":"eeaebb279140456c722be18222cfe34fe4c8f4092ef689e75a754e377382c219"} Oct 30 00:40:11.421296 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.421182 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p" event={"ID":"aadaf6fb-2702-4c04-88c9-cbb181049d68","Type":"ContainerDied","Data":"3214e1c7056e221cf7887decf7b678da2745a82a7faf17e64a695add10aab935"} Oct 30 00:40:11.421296 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.421204 5001 scope.go:122] "RemoveContainer" containerID="eeaebb279140456c722be18222cfe34fe4c8f4092ef689e75a754e377382c219" Oct 30 00:40:11.433426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.433399 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aadaf6fb-2702-4c04-88c9-cbb181049d68-kube-api-access-7kzzp\" (UniqueName: \"kubernetes.io/projected/aadaf6fb-2702-4c04-88c9-cbb181049d68-kube-api-access-7kzzp\") pod \"aadaf6fb-2702-4c04-88c9-cbb181049d68\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " Oct 30 00:40:11.433572 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.433448 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aadaf6fb-2702-4c04-88c9-cbb181049d68-istio-token\" (UniqueName: \"kubernetes.io/projected/aadaf6fb-2702-4c04-88c9-cbb181049d68-istio-token\") pod \"aadaf6fb-2702-4c04-88c9-cbb181049d68\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " Oct 30 00:40:11.433572 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.433486 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aadaf6fb-2702-4c04-88c9-cbb181049d68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aadaf6fb-2702-4c04-88c9-cbb181049d68-istiod-ca-cert\") pod \"aadaf6fb-2702-4c04-88c9-cbb181049d68\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " Oct 30 00:40:11.433572 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.433522 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-credential-socket\") pod \"aadaf6fb-2702-4c04-88c9-cbb181049d68\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " Oct 30 00:40:11.433760 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.433594 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-workload-certs\") pod \"aadaf6fb-2702-4c04-88c9-cbb181049d68\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " Oct 30 00:40:11.433760 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.433648 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-istio-data\") pod \"aadaf6fb-2702-4c04-88c9-cbb181049d68\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " Oct 30 00:40:11.433760 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.433693 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aadaf6fb-2702-4c04-88c9-cbb181049d68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aadaf6fb-2702-4c04-88c9-cbb181049d68-istio-podinfo\") pod \"aadaf6fb-2702-4c04-88c9-cbb181049d68\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " Oct 30 00:40:11.433760 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.433734 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-istio-envoy\") pod \"aadaf6fb-2702-4c04-88c9-cbb181049d68\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " Oct 30 00:40:11.434019 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.433768 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-workload-socket\") pod \"aadaf6fb-2702-4c04-88c9-cbb181049d68\" (UID: \"aadaf6fb-2702-4c04-88c9-cbb181049d68\") " Oct 30 00:40:11.434090 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.434010 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aadaf6fb-2702-4c04-88c9-cbb181049d68-istiod-ca-cert" pod "aadaf6fb-2702-4c04-88c9-cbb181049d68" (UID: "aadaf6fb-2702-4c04-88c9-cbb181049d68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:40:11.434261 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.434162 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-istio-data" pod "aadaf6fb-2702-4c04-88c9-cbb181049d68" (UID: "aadaf6fb-2702-4c04-88c9-cbb181049d68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:11.434345 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.434283 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-workload-socket" pod "aadaf6fb-2702-4c04-88c9-cbb181049d68" (UID: "aadaf6fb-2702-4c04-88c9-cbb181049d68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:11.434631 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.434606 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:40:11.434782 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.434766 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aadaf6fb-2702-4c04-88c9-cbb181049d68-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:40:11.435058 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.435035 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:40:11.435058 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.434769 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-credential-socket" pod "aadaf6fb-2702-4c04-88c9-cbb181049d68" (UID: "aadaf6fb-2702-4c04-88c9-cbb181049d68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:11.435296 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.434993 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-workload-certs" pod "aadaf6fb-2702-4c04-88c9-cbb181049d68" (UID: "aadaf6fb-2702-4c04-88c9-cbb181049d68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:11.435446 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.435403 5001 scope.go:122] "RemoveContainer" containerID="35460dd73d82bac9413321658766b2cade487258b6d5c905472184b20bbed8d6" Oct 30 00:40:11.436077 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.436042 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aadaf6fb-2702-4c04-88c9-cbb181049d68-istio-token" pod "aadaf6fb-2702-4c04-88c9-cbb181049d68" (UID: "aadaf6fb-2702-4c04-88c9-cbb181049d68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:40:11.436181 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.436163 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aadaf6fb-2702-4c04-88c9-cbb181049d68-kube-api-access-7kzzp" pod "aadaf6fb-2702-4c04-88c9-cbb181049d68" (UID: "aadaf6fb-2702-4c04-88c9-cbb181049d68"). InnerVolumeSpecName "kube-api-access-7kzzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:40:11.436654 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.436635 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aadaf6fb-2702-4c04-88c9-cbb181049d68-istio-podinfo" pod "aadaf6fb-2702-4c04-88c9-cbb181049d68" (UID: "aadaf6fb-2702-4c04-88c9-cbb181049d68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:40:11.436856 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.436837 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-istio-envoy" pod "aadaf6fb-2702-4c04-88c9-cbb181049d68" (UID: "aadaf6fb-2702-4c04-88c9-cbb181049d68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:11.457740 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.457718 5001 scope.go:122] "RemoveContainer" containerID="eeaebb279140456c722be18222cfe34fe4c8f4092ef689e75a754e377382c219" Oct 30 00:40:11.458086 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:40:11.458063 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeaebb279140456c722be18222cfe34fe4c8f4092ef689e75a754e377382c219\": container with ID starting with eeaebb279140456c722be18222cfe34fe4c8f4092ef689e75a754e377382c219 not found: ID does not exist" containerID="eeaebb279140456c722be18222cfe34fe4c8f4092ef689e75a754e377382c219" Oct 30 00:40:11.458168 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.458093 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeaebb279140456c722be18222cfe34fe4c8f4092ef689e75a754e377382c219"} err="failed to get container status \"eeaebb279140456c722be18222cfe34fe4c8f4092ef689e75a754e377382c219\": rpc error: code = NotFound desc = could not find container \"eeaebb279140456c722be18222cfe34fe4c8f4092ef689e75a754e377382c219\": container with ID starting with eeaebb279140456c722be18222cfe34fe4c8f4092ef689e75a754e377382c219 not found: ID does not exist" Oct 30 00:40:11.458168 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.458110 5001 scope.go:122] "RemoveContainer" containerID="35460dd73d82bac9413321658766b2cade487258b6d5c905472184b20bbed8d6" Oct 30 00:40:11.458376 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:40:11.458359 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35460dd73d82bac9413321658766b2cade487258b6d5c905472184b20bbed8d6\": container with ID starting with 35460dd73d82bac9413321658766b2cade487258b6d5c905472184b20bbed8d6 not found: ID does not exist" containerID="35460dd73d82bac9413321658766b2cade487258b6d5c905472184b20bbed8d6" Oct 30 00:40:11.458420 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.458383 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35460dd73d82bac9413321658766b2cade487258b6d5c905472184b20bbed8d6"} err="failed to get container status \"35460dd73d82bac9413321658766b2cade487258b6d5c905472184b20bbed8d6\": rpc error: code = NotFound desc = could not find container \"35460dd73d82bac9413321658766b2cade487258b6d5c905472184b20bbed8d6\": container with ID starting with 35460dd73d82bac9413321658766b2cade487258b6d5c905472184b20bbed8d6 not found: ID does not exist" Oct 30 00:40:11.536158 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.536123 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aadaf6fb-2702-4c04-88c9-cbb181049d68-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:40:11.536158 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.536154 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:40:11.536158 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.536164 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kzzp\" (UniqueName: \"kubernetes.io/projected/aadaf6fb-2702-4c04-88c9-cbb181049d68-kube-api-access-7kzzp\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:40:11.536376 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.536174 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aadaf6fb-2702-4c04-88c9-cbb181049d68-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:40:11.536376 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.536182 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:40:11.536376 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.536190 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aadaf6fb-2702-4c04-88c9-cbb181049d68-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:40:11.763063 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.763034 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p"] Oct 30 00:40:11.772216 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:11.772190 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-63e288-openshift-default-7f7d95db9f-69k7p"] Oct 30 00:40:12.917113 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:12.917078 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aadaf6fb-2702-4c04-88c9-cbb181049d68" path="/var/lib/kubelet/pods/aadaf6fb-2702-4c04-88c9-cbb181049d68/volumes" Oct 30 00:40:17.129289 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:17.129254 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f1268c-7745d464f5-hf7k2"] Oct 30 00:40:17.321768 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:40:17.321736 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:40:18.680072 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:18.680035 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f1268c-rhx28"] Oct 30 00:40:18.689586 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:18.689554 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f1268c-rhx28"] Oct 30 00:40:18.917343 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:18.917309 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53e4314e-f84a-4f80-8f61-d847f6425ff8" path="/var/lib/kubelet/pods/53e4314e-f84a-4f80-8f61-d847f6425ff8/volumes" Oct 30 00:40:22.129849 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:22.129778 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f1268c-7745d464f5-hf7k2" podUID="63af7822-1fdf-4af9-977b-11ce91f5ec63" containerName="payload-pre-processing" containerID="cri-o://390dcbf435e8ed1f319786b287313efb5259dae2a4cd01804706982bdc805bf1" gracePeriod=25 Oct 30 00:40:26.101142 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:26.101103 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh"] Oct 30 00:40:26.101641 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:26.101348 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" podUID="ff8dd458-3a31-4084-b71e-2ecb4114c4e0" containerName="istio-proxy" containerID="cri-o://8d892ea84ae30a92515fdd50f2e9fa4e8b07d234db4e759d796600fb8423bc0e" gracePeriod=30 Oct 30 00:40:27.143536 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:27.143499 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f1268c-7745d464f5-hf7k2" podUID="63af7822-1fdf-4af9-977b-11ce91f5ec63" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:40:31.360161 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.360133 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:40:31.538516 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.538481 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-workload-socket\") pod \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " Oct 30 00:40:31.538708 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.538552 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istio-token\" (UniqueName: \"kubernetes.io/projected/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istio-token\") pod \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " Oct 30 00:40:31.538708 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.538598 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-credential-socket\") pod \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " Oct 30 00:40:31.538800 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.538733 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-workload-certs\") pod \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " Oct 30 00:40:31.538800 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.538784 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istiod-ca-cert\") pod \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " Oct 30 00:40:31.538891 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.538815 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-kube-api-access-l7tb2\" (UniqueName: \"kubernetes.io/projected/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-kube-api-access-l7tb2\") pod \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " Oct 30 00:40:31.538891 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.538782 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-workload-socket" pod "ff8dd458-3a31-4084-b71e-2ecb4114c4e0" (UID: "ff8dd458-3a31-4084-b71e-2ecb4114c4e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:31.539030 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.538863 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istio-podinfo\") pod \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " Oct 30 00:40:31.539030 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.538895 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-credential-socket" pod "ff8dd458-3a31-4084-b71e-2ecb4114c4e0" (UID: "ff8dd458-3a31-4084-b71e-2ecb4114c4e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:31.539030 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.539005 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istio-envoy\") pod \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " Oct 30 00:40:31.539030 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.539005 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-workload-certs" pod "ff8dd458-3a31-4084-b71e-2ecb4114c4e0" (UID: "ff8dd458-3a31-4084-b71e-2ecb4114c4e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:31.539237 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.539070 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istio-data\") pod \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\" (UID: \"ff8dd458-3a31-4084-b71e-2ecb4114c4e0\") " Oct 30 00:40:31.539237 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.539112 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istiod-ca-cert" pod "ff8dd458-3a31-4084-b71e-2ecb4114c4e0" (UID: "ff8dd458-3a31-4084-b71e-2ecb4114c4e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:40:31.539461 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.539434 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:40:31.539461 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.539463 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:40:31.539697 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.539477 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:40:31.539697 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.539491 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:40:31.539697 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.539467 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istio-data" pod "ff8dd458-3a31-4084-b71e-2ecb4114c4e0" (UID: "ff8dd458-3a31-4084-b71e-2ecb4114c4e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:31.541024 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.540992 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istio-token" pod "ff8dd458-3a31-4084-b71e-2ecb4114c4e0" (UID: "ff8dd458-3a31-4084-b71e-2ecb4114c4e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:40:31.541024 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.541013 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istio-podinfo" pod "ff8dd458-3a31-4084-b71e-2ecb4114c4e0" (UID: "ff8dd458-3a31-4084-b71e-2ecb4114c4e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:40:31.541152 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.541040 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-kube-api-access-l7tb2" pod "ff8dd458-3a31-4084-b71e-2ecb4114c4e0" (UID: "ff8dd458-3a31-4084-b71e-2ecb4114c4e0"). InnerVolumeSpecName "kube-api-access-l7tb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:40:31.541152 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.541097 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istio-envoy" pod "ff8dd458-3a31-4084-b71e-2ecb4114c4e0" (UID: "ff8dd458-3a31-4084-b71e-2ecb4114c4e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:31.573293 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.573260 5001 generic.go:356] "Generic (PLEG): container finished" podID="ff8dd458-3a31-4084-b71e-2ecb4114c4e0" containerID="8d892ea84ae30a92515fdd50f2e9fa4e8b07d234db4e759d796600fb8423bc0e" exitCode=0 Oct 30 00:40:31.573426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.573318 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" Oct 30 00:40:31.573426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.573342 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" event={"ID":"ff8dd458-3a31-4084-b71e-2ecb4114c4e0","Type":"ContainerDied","Data":"8d892ea84ae30a92515fdd50f2e9fa4e8b07d234db4e759d796600fb8423bc0e"} Oct 30 00:40:31.573426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.573397 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh" event={"ID":"ff8dd458-3a31-4084-b71e-2ecb4114c4e0","Type":"ContainerDied","Data":"13d4a1291df03ed72e65fc26db22c36d56283ed8b4750fa62f2a50beaae39b8a"} Oct 30 00:40:31.573426 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.573420 5001 scope.go:122] "RemoveContainer" containerID="8d892ea84ae30a92515fdd50f2e9fa4e8b07d234db4e759d796600fb8423bc0e" Oct 30 00:40:31.585110 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.585085 5001 scope.go:122] "RemoveContainer" containerID="90b44cb7797ab9211767e050a7f01e06de111cf75c35871faa72b7c2e1f8587e" Oct 30 00:40:31.595905 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.595878 5001 scope.go:122] "RemoveContainer" containerID="8d892ea84ae30a92515fdd50f2e9fa4e8b07d234db4e759d796600fb8423bc0e" Oct 30 00:40:31.596078 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.596057 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp"] Oct 30 00:40:31.596201 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:40:31.596181 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d892ea84ae30a92515fdd50f2e9fa4e8b07d234db4e759d796600fb8423bc0e\": container with ID starting with 8d892ea84ae30a92515fdd50f2e9fa4e8b07d234db4e759d796600fb8423bc0e not found: ID does not exist" containerID="8d892ea84ae30a92515fdd50f2e9fa4e8b07d234db4e759d796600fb8423bc0e" Oct 30 00:40:31.596268 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.596212 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d892ea84ae30a92515fdd50f2e9fa4e8b07d234db4e759d796600fb8423bc0e"} err="failed to get container status \"8d892ea84ae30a92515fdd50f2e9fa4e8b07d234db4e759d796600fb8423bc0e\": rpc error: code = NotFound desc = could not find container \"8d892ea84ae30a92515fdd50f2e9fa4e8b07d234db4e759d796600fb8423bc0e\": container with ID starting with 8d892ea84ae30a92515fdd50f2e9fa4e8b07d234db4e759d796600fb8423bc0e not found: ID does not exist" Oct 30 00:40:31.596268 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.596237 5001 scope.go:122] "RemoveContainer" containerID="90b44cb7797ab9211767e050a7f01e06de111cf75c35871faa72b7c2e1f8587e" Oct 30 00:40:31.596550 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:40:31.596531 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90b44cb7797ab9211767e050a7f01e06de111cf75c35871faa72b7c2e1f8587e\": container with ID starting with 90b44cb7797ab9211767e050a7f01e06de111cf75c35871faa72b7c2e1f8587e not found: ID does not exist" containerID="90b44cb7797ab9211767e050a7f01e06de111cf75c35871faa72b7c2e1f8587e" Oct 30 00:40:31.596613 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.596557 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b44cb7797ab9211767e050a7f01e06de111cf75c35871faa72b7c2e1f8587e"} err="failed to get container status \"90b44cb7797ab9211767e050a7f01e06de111cf75c35871faa72b7c2e1f8587e\": rpc error: code = NotFound desc = could not find container \"90b44cb7797ab9211767e050a7f01e06de111cf75c35871faa72b7c2e1f8587e\": container with ID starting with 90b44cb7797ab9211767e050a7f01e06de111cf75c35871faa72b7c2e1f8587e not found: ID does not exist" Oct 30 00:40:31.631848 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.631823 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh"] Oct 30 00:40:31.638322 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.638298 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f1268c-openshift-default-6589bc648f-5c6dh"] Oct 30 00:40:31.640278 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.640257 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7tb2\" (UniqueName: \"kubernetes.io/projected/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-kube-api-access-l7tb2\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:40:31.640348 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.640280 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:40:31.640348 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.640291 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:40:31.640348 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.640299 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:40:31.640348 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:31.640307 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff8dd458-3a31-4084-b71e-2ecb4114c4e0-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:40:32.917223 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:32.917187 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff8dd458-3a31-4084-b71e-2ecb4114c4e0" path="/var/lib/kubelet/pods/ff8dd458-3a31-4084-b71e-2ecb4114c4e0/volumes" Oct 30 00:40:37.143855 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:37.143817 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f1268c-7745d464f5-hf7k2" podUID="63af7822-1fdf-4af9-977b-11ce91f5ec63" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:40:37.257139 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:37.257100 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-6c45bb-546469c9db-sn7gc"] Oct 30 00:40:38.699256 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:38.699218 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-778941da-sljf5"] Oct 30 00:40:38.699674 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:38.699621 5001 cpu_manager.go:414] "containerMap: removing container" podUID="aadaf6fb-2702-4c04-88c9-cbb181049d68" containerName="istio-proxy" Oct 30 00:40:38.699674 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:38.699633 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aadaf6fb-2702-4c04-88c9-cbb181049d68" containerName="istio-proxy" Oct 30 00:40:38.699674 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:38.699643 5001 cpu_manager.go:414] "containerMap: removing container" podUID="ff8dd458-3a31-4084-b71e-2ecb4114c4e0" containerName="istio-proxy" Oct 30 00:40:38.699674 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:38.699648 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff8dd458-3a31-4084-b71e-2ecb4114c4e0" containerName="istio-proxy" Oct 30 00:40:38.699674 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:38.699659 5001 cpu_manager.go:414] "containerMap: removing container" podUID="ff8dd458-3a31-4084-b71e-2ecb4114c4e0" containerName="istio-proxy" Oct 30 00:40:38.699674 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:38.699663 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff8dd458-3a31-4084-b71e-2ecb4114c4e0" containerName="istio-proxy" Oct 30 00:40:38.699674 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:38.699673 5001 cpu_manager.go:414] "containerMap: removing container" podUID="53e4314e-f84a-4f80-8f61-d847f6425ff8" containerName="revoke-keys" Oct 30 00:40:38.699674 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:38.699678 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53e4314e-f84a-4f80-8f61-d847f6425ff8" containerName="revoke-keys" Oct 30 00:40:38.699971 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:38.699693 5001 cpu_manager.go:414] "containerMap: removing container" podUID="6cad4256-1e03-4119-9cbe-4dbe1717600a" containerName="istio-proxy" Oct 30 00:40:38.699971 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:38.699699 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cad4256-1e03-4119-9cbe-4dbe1717600a" containerName="istio-proxy" Oct 30 00:40:38.699971 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:38.699708 5001 cpu_manager.go:414] "containerMap: removing container" podUID="aadaf6fb-2702-4c04-88c9-cbb181049d68" containerName="istio-proxy" Oct 30 00:40:38.699971 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:38.699713 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aadaf6fb-2702-4c04-88c9-cbb181049d68" containerName="istio-proxy" Oct 30 00:40:38.699971 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:38.699775 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="aadaf6fb-2702-4c04-88c9-cbb181049d68" containerName="istio-proxy" Oct 30 00:40:38.699971 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:38.699783 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff8dd458-3a31-4084-b71e-2ecb4114c4e0" containerName="istio-proxy" Oct 30 00:40:38.699971 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:38.699788 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cad4256-1e03-4119-9cbe-4dbe1717600a" containerName="istio-proxy" Oct 30 00:40:38.699971 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:38.699797 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="53e4314e-f84a-4f80-8f61-d847f6425ff8" containerName="revoke-keys" Oct 30 00:40:38.704342 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:38.704325 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-778941da-sljf5" Oct 30 00:40:38.729157 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:38.729135 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:40:38.729322 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:38.729135 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:40:38.729453 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:38.729199 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:40:38.729453 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:38.729265 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2t9px\"" Oct 30 00:40:38.732871 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:38.732847 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-778941da-sljf5"] Oct 30 00:40:38.811284 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:38.811255 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/516e9889-a327-46ed-90f1-59189277128c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-778941da-sljf5\" (UID: \"516e9889-a327-46ed-90f1-59189277128c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-778941da-sljf5" Oct 30 00:40:38.811429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:38.811307 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/516e9889-a327-46ed-90f1-59189277128c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-778941da-sljf5\" (UID: \"516e9889-a327-46ed-90f1-59189277128c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-778941da-sljf5" Oct 30 00:40:38.811495 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:38.811440 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnw85\" (UniqueName: \"kubernetes.io/projected/516e9889-a327-46ed-90f1-59189277128c-kube-api-access-fnw85\") pod \"maas-api-revoke-sub-778941da-sljf5\" (UID: \"516e9889-a327-46ed-90f1-59189277128c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-778941da-sljf5" Oct 30 00:40:38.912572 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:38.912541 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnw85\" (UniqueName: \"kubernetes.io/projected/516e9889-a327-46ed-90f1-59189277128c-kube-api-access-fnw85\") pod \"maas-api-revoke-sub-778941da-sljf5\" (UID: \"516e9889-a327-46ed-90f1-59189277128c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-778941da-sljf5" Oct 30 00:40:38.912739 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:38.912606 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/516e9889-a327-46ed-90f1-59189277128c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-778941da-sljf5\" (UID: \"516e9889-a327-46ed-90f1-59189277128c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-778941da-sljf5" Oct 30 00:40:38.912739 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:38.912654 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/516e9889-a327-46ed-90f1-59189277128c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-778941da-sljf5\" (UID: \"516e9889-a327-46ed-90f1-59189277128c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-778941da-sljf5" Oct 30 00:40:38.913323 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:38.913299 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/516e9889-a327-46ed-90f1-59189277128c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-778941da-sljf5\" (UID: \"516e9889-a327-46ed-90f1-59189277128c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-778941da-sljf5" Oct 30 00:40:38.915198 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:38.915179 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/516e9889-a327-46ed-90f1-59189277128c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-778941da-sljf5\" (UID: \"516e9889-a327-46ed-90f1-59189277128c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-778941da-sljf5" Oct 30 00:40:38.932990 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:38.932968 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnw85\" (UniqueName: \"kubernetes.io/projected/516e9889-a327-46ed-90f1-59189277128c-kube-api-access-fnw85\") pod \"maas-api-revoke-sub-778941da-sljf5\" (UID: \"516e9889-a327-46ed-90f1-59189277128c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-778941da-sljf5" Oct 30 00:40:39.018257 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:39.018226 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-778941da-sljf5" Oct 30 00:40:39.318765 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:39.318734 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-778941da-sljf5"] Oct 30 00:40:39.319596 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:40:39.319564 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod516e9889_a327_46ed_90f1_59189277128c.slice/crio-ebe7aafc5662dfbf300652f5e9c498beedf948e344cadeaf17db7e93292913e1 WatchSource:0}: Error finding container ebe7aafc5662dfbf300652f5e9c498beedf948e344cadeaf17db7e93292913e1: Status 404 returned error can't find the container with id ebe7aafc5662dfbf300652f5e9c498beedf948e344cadeaf17db7e93292913e1 Oct 30 00:40:39.632354 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:39.632321 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-778941da-sljf5" event={"ID":"516e9889-a327-46ed-90f1-59189277128c","Type":"ContainerStarted","Data":"f90114fb30e4ba6b108032e4c5f46535ba546cd50af91c071562b1eea0aec6a0"} Oct 30 00:40:39.632486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:39.632371 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-778941da-sljf5" event={"ID":"516e9889-a327-46ed-90f1-59189277128c","Type":"ContainerStarted","Data":"ebe7aafc5662dfbf300652f5e9c498beedf948e344cadeaf17db7e93292913e1"} Oct 30 00:40:39.670362 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:39.670311 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-778941da-sljf5" podStartSLOduration=1.670298796 podStartE2EDuration="1.670298796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:40:39.668693481 +0000 UTC m=+2154.874194663" watchObservedRunningTime="2026-10-30 00:40:39.670298796 +0000 UTC m=+2154.875799976" Oct 30 00:40:40.648180 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:40.648149 5001 generic.go:356] "Generic (PLEG): container finished" podID="516e9889-a327-46ed-90f1-59189277128c" containerID="f90114fb30e4ba6b108032e4c5f46535ba546cd50af91c071562b1eea0aec6a0" exitCode=0 Oct 30 00:40:40.648584 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:40.648231 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-778941da-sljf5" event={"ID":"516e9889-a327-46ed-90f1-59189277128c","Type":"ContainerDied","Data":"f90114fb30e4ba6b108032e4c5f46535ba546cd50af91c071562b1eea0aec6a0"} Oct 30 00:40:41.792776 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:41.792747 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-778941da-sljf5" Oct 30 00:40:41.840612 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:41.840532 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/516e9889-a327-46ed-90f1-59189277128c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/516e9889-a327-46ed-90f1-59189277128c-maas-api-openshift-ca\") pod \"516e9889-a327-46ed-90f1-59189277128c\" (UID: \"516e9889-a327-46ed-90f1-59189277128c\") " Oct 30 00:40:41.840784 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:41.840689 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/516e9889-a327-46ed-90f1-59189277128c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/516e9889-a327-46ed-90f1-59189277128c-maas-api-xks-ca\") pod \"516e9889-a327-46ed-90f1-59189277128c\" (UID: \"516e9889-a327-46ed-90f1-59189277128c\") " Oct 30 00:40:41.840784 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:41.840744 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/516e9889-a327-46ed-90f1-59189277128c-kube-api-access-fnw85\" (UniqueName: \"kubernetes.io/projected/516e9889-a327-46ed-90f1-59189277128c-kube-api-access-fnw85\") pod \"516e9889-a327-46ed-90f1-59189277128c\" (UID: \"516e9889-a327-46ed-90f1-59189277128c\") " Oct 30 00:40:41.840964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:41.840936 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/516e9889-a327-46ed-90f1-59189277128c-maas-api-openshift-ca" pod "516e9889-a327-46ed-90f1-59189277128c" (UID: "516e9889-a327-46ed-90f1-59189277128c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:40:41.841165 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:41.841139 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/516e9889-a327-46ed-90f1-59189277128c-maas-api-openshift-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:40:41.843477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:41.843446 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/516e9889-a327-46ed-90f1-59189277128c-maas-api-xks-ca" pod "516e9889-a327-46ed-90f1-59189277128c" (UID: "516e9889-a327-46ed-90f1-59189277128c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:40:41.843477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:41.843462 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/516e9889-a327-46ed-90f1-59189277128c-kube-api-access-fnw85" pod "516e9889-a327-46ed-90f1-59189277128c" (UID: "516e9889-a327-46ed-90f1-59189277128c"). InnerVolumeSpecName "kube-api-access-fnw85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:40:41.942471 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:41.942390 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/516e9889-a327-46ed-90f1-59189277128c-maas-api-xks-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:40:41.942471 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:41.942424 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnw85\" (UniqueName: \"kubernetes.io/projected/516e9889-a327-46ed-90f1-59189277128c-kube-api-access-fnw85\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:40:42.258297 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:42.258257 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-6c45bb-546469c9db-sn7gc" podUID="e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e" containerName="payload-processing" containerID="cri-o://14a7d2d61ae1c3af08156d67e12b3f5280552cc84503079efd3111c2dc380b0f" gracePeriod=25 Oct 30 00:40:42.370260 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:42.370232 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f1268c-7745d464f5-hf7k2" Oct 30 00:40:42.446595 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:42.446555 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63af7822-1fdf-4af9-977b-11ce91f5ec63-kube-api-access-qxvxd\" (UniqueName: \"kubernetes.io/projected/63af7822-1fdf-4af9-977b-11ce91f5ec63-kube-api-access-qxvxd\") pod \"63af7822-1fdf-4af9-977b-11ce91f5ec63\" (UID: \"63af7822-1fdf-4af9-977b-11ce91f5ec63\") " Oct 30 00:40:42.446770 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:42.446627 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63af7822-1fdf-4af9-977b-11ce91f5ec63-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63af7822-1fdf-4af9-977b-11ce91f5ec63-plugins-config-volume\") pod \"63af7822-1fdf-4af9-977b-11ce91f5ec63\" (UID: \"63af7822-1fdf-4af9-977b-11ce91f5ec63\") " Oct 30 00:40:42.447112 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:42.447089 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63af7822-1fdf-4af9-977b-11ce91f5ec63-plugins-config-volume" pod "63af7822-1fdf-4af9-977b-11ce91f5ec63" (UID: "63af7822-1fdf-4af9-977b-11ce91f5ec63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:40:42.448836 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:42.448813 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63af7822-1fdf-4af9-977b-11ce91f5ec63-kube-api-access-qxvxd" pod "63af7822-1fdf-4af9-977b-11ce91f5ec63" (UID: "63af7822-1fdf-4af9-977b-11ce91f5ec63"). InnerVolumeSpecName "kube-api-access-qxvxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:40:42.548388 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:42.548348 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxvxd\" (UniqueName: \"kubernetes.io/projected/63af7822-1fdf-4af9-977b-11ce91f5ec63-kube-api-access-qxvxd\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:40:42.548388 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:42.548380 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63af7822-1fdf-4af9-977b-11ce91f5ec63-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:40:42.663771 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:42.663734 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-778941da-sljf5" Oct 30 00:40:42.663975 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:42.663771 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-778941da-sljf5" event={"ID":"516e9889-a327-46ed-90f1-59189277128c","Type":"ContainerDied","Data":"ebe7aafc5662dfbf300652f5e9c498beedf948e344cadeaf17db7e93292913e1"} Oct 30 00:40:42.663975 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:42.663833 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebe7aafc5662dfbf300652f5e9c498beedf948e344cadeaf17db7e93292913e1" Oct 30 00:40:42.665480 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:42.665440 5001 generic.go:356] "Generic (PLEG): container finished" podID="63af7822-1fdf-4af9-977b-11ce91f5ec63" containerID="390dcbf435e8ed1f319786b287313efb5259dae2a4cd01804706982bdc805bf1" exitCode=0 Oct 30 00:40:42.665480 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:42.665474 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f1268c-7745d464f5-hf7k2" Oct 30 00:40:42.665679 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:42.665475 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f1268c-7745d464f5-hf7k2" event={"ID":"63af7822-1fdf-4af9-977b-11ce91f5ec63","Type":"ContainerDied","Data":"390dcbf435e8ed1f319786b287313efb5259dae2a4cd01804706982bdc805bf1"} Oct 30 00:40:42.665679 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:42.665598 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f1268c-7745d464f5-hf7k2" event={"ID":"63af7822-1fdf-4af9-977b-11ce91f5ec63","Type":"ContainerDied","Data":"4fe68158eaa7ab62b1392c713addc8ed533e5df47220b66c551ef51ca3c22890"} Oct 30 00:40:42.665679 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:42.665627 5001 scope.go:122] "RemoveContainer" containerID="390dcbf435e8ed1f319786b287313efb5259dae2a4cd01804706982bdc805bf1" Oct 30 00:40:42.677307 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:42.677285 5001 scope.go:122] "RemoveContainer" containerID="390dcbf435e8ed1f319786b287313efb5259dae2a4cd01804706982bdc805bf1" Oct 30 00:40:42.677616 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:40:42.677589 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"390dcbf435e8ed1f319786b287313efb5259dae2a4cd01804706982bdc805bf1\": container with ID starting with 390dcbf435e8ed1f319786b287313efb5259dae2a4cd01804706982bdc805bf1 not found: ID does not exist" containerID="390dcbf435e8ed1f319786b287313efb5259dae2a4cd01804706982bdc805bf1" Oct 30 00:40:42.677709 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:42.677624 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"390dcbf435e8ed1f319786b287313efb5259dae2a4cd01804706982bdc805bf1"} err="failed to get container status \"390dcbf435e8ed1f319786b287313efb5259dae2a4cd01804706982bdc805bf1\": rpc error: code = NotFound desc = could not find container \"390dcbf435e8ed1f319786b287313efb5259dae2a4cd01804706982bdc805bf1\": container with ID starting with 390dcbf435e8ed1f319786b287313efb5259dae2a4cd01804706982bdc805bf1 not found: ID does not exist" Oct 30 00:40:42.711234 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:42.711200 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f1268c-7745d464f5-hf7k2"] Oct 30 00:40:42.714691 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:42.714665 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f1268c-7745d464f5-hf7k2"] Oct 30 00:40:42.916902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:42.916825 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63af7822-1fdf-4af9-977b-11ce91f5ec63" path="/var/lib/kubelet/pods/63af7822-1fdf-4af9-977b-11ce91f5ec63/volumes" Oct 30 00:40:46.618122 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:46.618050 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" containerID="cri-o://a262e489e843f3476800ff87eac5543a3377c42dd3509865ef8b2c49c30f7201" gracePeriod=45 Oct 30 00:40:46.872056 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:46.871988 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:40:46.986965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:46.986932 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c7951f-8758-4b9c-b91a-00024f9bf582-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0c7951f-8758-4b9c-b91a-00024f9bf582-dshm\") pod \"b0c7951f-8758-4b9c-b91a-00024f9bf582\" (UID: \"b0c7951f-8758-4b9c-b91a-00024f9bf582\") " Oct 30 00:40:46.987168 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:46.986987 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0c7951f-8758-4b9c-b91a-00024f9bf582-kube-api-access-g26qv\" (UniqueName: \"kubernetes.io/projected/b0c7951f-8758-4b9c-b91a-00024f9bf582-kube-api-access-g26qv\") pod \"b0c7951f-8758-4b9c-b91a-00024f9bf582\" (UID: \"b0c7951f-8758-4b9c-b91a-00024f9bf582\") " Oct 30 00:40:46.987168 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:46.987041 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c7951f-8758-4b9c-b91a-00024f9bf582-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0c7951f-8758-4b9c-b91a-00024f9bf582-model-cache\") pod \"b0c7951f-8758-4b9c-b91a-00024f9bf582\" (UID: \"b0c7951f-8758-4b9c-b91a-00024f9bf582\") " Oct 30 00:40:46.987168 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:46.987071 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c7951f-8758-4b9c-b91a-00024f9bf582-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0c7951f-8758-4b9c-b91a-00024f9bf582-tmp-dir\") pod \"b0c7951f-8758-4b9c-b91a-00024f9bf582\" (UID: \"b0c7951f-8758-4b9c-b91a-00024f9bf582\") " Oct 30 00:40:46.987168 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:46.987107 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c7951f-8758-4b9c-b91a-00024f9bf582-home\" (UniqueName: \"kubernetes.io/empty-dir/b0c7951f-8758-4b9c-b91a-00024f9bf582-home\") pod \"b0c7951f-8758-4b9c-b91a-00024f9bf582\" (UID: \"b0c7951f-8758-4b9c-b91a-00024f9bf582\") " Oct 30 00:40:46.987400 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:46.987178 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0c7951f-8758-4b9c-b91a-00024f9bf582-tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c7951f-8758-4b9c-b91a-00024f9bf582-tls-certs\") pod \"b0c7951f-8758-4b9c-b91a-00024f9bf582\" (UID: \"b0c7951f-8758-4b9c-b91a-00024f9bf582\") " Oct 30 00:40:46.987400 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:46.987356 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c7951f-8758-4b9c-b91a-00024f9bf582-model-cache" pod "b0c7951f-8758-4b9c-b91a-00024f9bf582" (UID: "b0c7951f-8758-4b9c-b91a-00024f9bf582"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:46.987400 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:46.987369 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c7951f-8758-4b9c-b91a-00024f9bf582-tmp-dir" pod "b0c7951f-8758-4b9c-b91a-00024f9bf582" (UID: "b0c7951f-8758-4b9c-b91a-00024f9bf582"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:46.987533 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:46.987478 5001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0c7951f-8758-4b9c-b91a-00024f9bf582-model-cache\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:40:46.987533 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:46.987521 5001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0c7951f-8758-4b9c-b91a-00024f9bf582-tmp-dir\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:40:46.987837 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:46.987808 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c7951f-8758-4b9c-b91a-00024f9bf582-home" pod "b0c7951f-8758-4b9c-b91a-00024f9bf582" (UID: "b0c7951f-8758-4b9c-b91a-00024f9bf582"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:46.989399 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:46.989367 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c7951f-8758-4b9c-b91a-00024f9bf582-kube-api-access-g26qv" pod "b0c7951f-8758-4b9c-b91a-00024f9bf582" (UID: "b0c7951f-8758-4b9c-b91a-00024f9bf582"). InnerVolumeSpecName "kube-api-access-g26qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:40:46.989399 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:46.989384 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c7951f-8758-4b9c-b91a-00024f9bf582-dshm" pod "b0c7951f-8758-4b9c-b91a-00024f9bf582" (UID: "b0c7951f-8758-4b9c-b91a-00024f9bf582"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:46.989555 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:46.989456 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0c7951f-8758-4b9c-b91a-00024f9bf582-tls-certs" pod "b0c7951f-8758-4b9c-b91a-00024f9bf582" (UID: "b0c7951f-8758-4b9c-b91a-00024f9bf582"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:40:47.088445 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:47.088409 5001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0c7951f-8758-4b9c-b91a-00024f9bf582-dshm\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:40:47.088445 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:47.088443 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g26qv\" (UniqueName: \"kubernetes.io/projected/b0c7951f-8758-4b9c-b91a-00024f9bf582-kube-api-access-g26qv\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:40:47.088664 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:47.088456 5001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0c7951f-8758-4b9c-b91a-00024f9bf582-home\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:40:47.088664 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:47.088467 5001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c7951f-8758-4b9c-b91a-00024f9bf582-tls-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:40:47.321796 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:40:47.321756 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:40:47.703204 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:47.703110 5001 generic.go:356] "Generic (PLEG): container finished" podID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerID="a262e489e843f3476800ff87eac5543a3377c42dd3509865ef8b2c49c30f7201" exitCode=0 Oct 30 00:40:47.703204 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:47.703164 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" Oct 30 00:40:47.703700 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:47.703174 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" event={"ID":"b0c7951f-8758-4b9c-b91a-00024f9bf582","Type":"ContainerDied","Data":"a262e489e843f3476800ff87eac5543a3377c42dd3509865ef8b2c49c30f7201"} Oct 30 00:40:47.703700 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:47.703277 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp" event={"ID":"b0c7951f-8758-4b9c-b91a-00024f9bf582","Type":"ContainerDied","Data":"a2e0834bac1fa769b4eb430be3fe17de567d7f79a7d7b71ffb32ce3fcaa47d6b"} Oct 30 00:40:47.703700 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:47.703295 5001 scope.go:122] "RemoveContainer" containerID="a262e489e843f3476800ff87eac5543a3377c42dd3509865ef8b2c49c30f7201" Oct 30 00:40:47.713987 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:47.713965 5001 scope.go:122] "RemoveContainer" containerID="a262e489e843f3476800ff87eac5543a3377c42dd3509865ef8b2c49c30f7201" Oct 30 00:40:47.714290 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:40:47.714269 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a262e489e843f3476800ff87eac5543a3377c42dd3509865ef8b2c49c30f7201\": container with ID starting with a262e489e843f3476800ff87eac5543a3377c42dd3509865ef8b2c49c30f7201 not found: ID does not exist" containerID="a262e489e843f3476800ff87eac5543a3377c42dd3509865ef8b2c49c30f7201" Oct 30 00:40:47.714337 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:47.714301 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a262e489e843f3476800ff87eac5543a3377c42dd3509865ef8b2c49c30f7201"} err="failed to get container status \"a262e489e843f3476800ff87eac5543a3377c42dd3509865ef8b2c49c30f7201\": rpc error: code = NotFound desc = could not find container \"a262e489e843f3476800ff87eac5543a3377c42dd3509865ef8b2c49c30f7201\": container with ID starting with a262e489e843f3476800ff87eac5543a3377c42dd3509865ef8b2c49c30f7201 not found: ID does not exist" Oct 30 00:40:47.748486 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:47.748454 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp"] Oct 30 00:40:47.758148 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:47.758121 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-6c45bb/simulator-w6-6c45bb-kserve-7cd6bb84fc-5zrfp"] Oct 30 00:40:48.916483 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:48.916449 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" path="/var/lib/kubelet/pods/b0c7951f-8758-4b9c-b91a-00024f9bf582/volumes" Oct 30 00:40:50.643340 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:40:50.643301 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-6c45bb-546469c9db-sn7gc" podUID="e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:41:00.643229 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:00.643187 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-6c45bb-546469c9db-sn7gc" podUID="e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:41:01.058152 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:01.058115 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675"] Oct 30 00:41:01.058452 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:01.058423 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" podUID="b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" containerName="istio-proxy" containerID="cri-o://e42b653a6ed631241bde2d9d65f58d134b03524efc2d224f5e8766e5fff9044c" gracePeriod=30 Oct 30 00:41:01.065315 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:01.065289 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl"] Oct 30 00:41:01.065540 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:01.065516 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" podUID="bfd5b4b9-3358-4549-80ba-5b84960fd3c0" containerName="istio-proxy" containerID="cri-o://1e2e0a73b6b9055d8b01356a1d7027e4571dd7d1d26adcb0f4cc8fb5ecf352a0" gracePeriod=30 Oct 30 00:41:01.940548 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:01.940511 5001 scope.go:122] "RemoveContainer" containerID="6b18862add77c626a7a42122b49ed976bb87da447170e9917c4dcc9df449ee7f" Oct 30 00:41:01.950318 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:01.950296 5001 scope.go:122] "RemoveContainer" containerID="7db5d89a112e12eba811e2bbf825c08632fd0e712d7f6c943a219811976d0e80" Oct 30 00:41:01.958956 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:01.958938 5001 scope.go:122] "RemoveContainer" containerID="76a93fc95f464804448af3c16a6ffcec1dedc2d80aaaea70d45ea9913a38c695" Oct 30 00:41:02.510173 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:02.510151 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-6c45bb-546469c9db-sn7gc" Oct 30 00:41:02.642702 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:02.642625 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e-kube-api-access-w9bnf\" (UniqueName: \"kubernetes.io/projected/e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e-kube-api-access-w9bnf\") pod \"e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e\" (UID: \"e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e\") " Oct 30 00:41:02.642702 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:02.642689 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e-plugins-config-volume\") pod \"e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e\" (UID: \"e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e\") " Oct 30 00:41:02.643092 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:02.643067 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e-plugins-config-volume" pod "e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e" (UID: "e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:41:02.644802 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:02.644778 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e-kube-api-access-w9bnf" pod "e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e" (UID: "e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e"). InnerVolumeSpecName "kube-api-access-w9bnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:41:02.676837 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:02.676810 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-vxqwt_0aa4310d-ba64-4c3a-b1ca-0d5a8d1522fb/manager/0.log" Oct 30 00:41:02.744049 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:02.744021 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9bnf\" (UniqueName: \"kubernetes.io/projected/e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e-kube-api-access-w9bnf\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:41:02.744049 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:02.744046 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:41:02.813084 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:02.813050 5001 generic.go:356] "Generic (PLEG): container finished" podID="e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e" containerID="14a7d2d61ae1c3af08156d67e12b3f5280552cc84503079efd3111c2dc380b0f" exitCode=0 Oct 30 00:41:02.813262 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:02.813111 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-6c45bb-546469c9db-sn7gc" Oct 30 00:41:02.813262 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:02.813124 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-6c45bb-546469c9db-sn7gc" event={"ID":"e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e","Type":"ContainerDied","Data":"14a7d2d61ae1c3af08156d67e12b3f5280552cc84503079efd3111c2dc380b0f"} Oct 30 00:41:02.813262 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:02.813182 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-6c45bb-546469c9db-sn7gc" event={"ID":"e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e","Type":"ContainerDied","Data":"25ce406282dd1241f8889d9a3d226453ef043e2bf8a43d1b1a27ff3d2103de90"} Oct 30 00:41:02.813262 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:02.813214 5001 scope.go:122] "RemoveContainer" containerID="14a7d2d61ae1c3af08156d67e12b3f5280552cc84503079efd3111c2dc380b0f" Oct 30 00:41:02.823797 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:02.823781 5001 scope.go:122] "RemoveContainer" containerID="14a7d2d61ae1c3af08156d67e12b3f5280552cc84503079efd3111c2dc380b0f" Oct 30 00:41:02.824094 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:41:02.824076 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a7d2d61ae1c3af08156d67e12b3f5280552cc84503079efd3111c2dc380b0f\": container with ID starting with 14a7d2d61ae1c3af08156d67e12b3f5280552cc84503079efd3111c2dc380b0f not found: ID does not exist" containerID="14a7d2d61ae1c3af08156d67e12b3f5280552cc84503079efd3111c2dc380b0f" Oct 30 00:41:02.824160 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:02.824103 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a7d2d61ae1c3af08156d67e12b3f5280552cc84503079efd3111c2dc380b0f"} err="failed to get container status \"14a7d2d61ae1c3af08156d67e12b3f5280552cc84503079efd3111c2dc380b0f\": rpc error: code = NotFound desc = could not find container \"14a7d2d61ae1c3af08156d67e12b3f5280552cc84503079efd3111c2dc380b0f\": container with ID starting with 14a7d2d61ae1c3af08156d67e12b3f5280552cc84503079efd3111c2dc380b0f not found: ID does not exist" Oct 30 00:41:02.857491 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:02.857467 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-6c45bb-546469c9db-sn7gc"] Oct 30 00:41:02.860759 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:02.860738 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-6c45bb-546469c9db-sn7gc"] Oct 30 00:41:02.880339 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:02.880319 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-z72x7_449bf72c-2ac6-42f7-8174-1e51e4c792cc/kuadrant-console-plugin/0.log" Oct 30 00:41:02.916978 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:02.916895 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e" path="/var/lib/kubelet/pods/e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e/volumes" Oct 30 00:41:03.624447 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:03.624399 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-xr4lb_75b06ad5-f34b-4a66-b00a-50325ca9fc0b/manager/0.log" Oct 30 00:41:05.176755 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:05.176722 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-4n6qq_3febb7a0-47fa-4545-b76c-977789498353/manager/1.log" Oct 30 00:41:05.412823 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:05.412789 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-p564d_f408e652-4069-4d1d-b5b0-abc34593794c/manager/0.log" Oct 30 00:41:06.317821 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.317794 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:41:06.349989 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.349967 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:41:06.475765 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.475734 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-kube-api-access-xcxnd\" (UniqueName: \"kubernetes.io/projected/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-kube-api-access-xcxnd\") pod \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " Oct 30 00:41:06.475963 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.475776 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istio-token\" (UniqueName: \"kubernetes.io/projected/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istio-token\") pod \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " Oct 30 00:41:06.475963 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.475800 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-workload-socket\") pod \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " Oct 30 00:41:06.475963 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.475820 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-kube-api-access-zhdxt\" (UniqueName: \"kubernetes.io/projected/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-kube-api-access-zhdxt\") pod \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " Oct 30 00:41:06.475963 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.475869 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istio-envoy\") pod \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " Oct 30 00:41:06.475963 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.475890 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istio-data\") pod \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " Oct 30 00:41:06.475963 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.475953 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-workload-certs\") pod \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " Oct 30 00:41:06.476306 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.476112 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-workload-socket" pod "b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" (UID: "b5e89f70-0dab-4ed5-b01b-2e78feab8b8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:41:06.476306 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.476226 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-workload-certs" pod "bfd5b4b9-3358-4549-80ba-5b84960fd3c0" (UID: "bfd5b4b9-3358-4549-80ba-5b84960fd3c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:41:06.476306 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.476239 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istio-podinfo\") pod \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " Oct 30 00:41:06.476306 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.476291 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-credential-socket\") pod \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " Oct 30 00:41:06.476528 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.476338 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istio-envoy\") pod \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " Oct 30 00:41:06.476528 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.476341 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istio-data" pod "b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" (UID: "b5e89f70-0dab-4ed5-b01b-2e78feab8b8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:41:06.476528 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.476381 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-workload-socket\") pod \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " Oct 30 00:41:06.476528 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.476412 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istio-data\") pod \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " Oct 30 00:41:06.476528 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.476442 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-credential-socket\") pod \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " Oct 30 00:41:06.476528 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.476479 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-workload-certs\") pod \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " Oct 30 00:41:06.476528 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.476518 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istiod-ca-cert\") pod \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " Oct 30 00:41:06.476900 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.476548 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istiod-ca-cert\") pod \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " Oct 30 00:41:06.476900 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.476591 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istio-token\" (UniqueName: \"kubernetes.io/projected/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istio-token\") pod \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\" (UID: \"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e\") " Oct 30 00:41:06.476900 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.476631 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istio-podinfo\") pod \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\" (UID: \"bfd5b4b9-3358-4549-80ba-5b84960fd3c0\") " Oct 30 00:41:06.476900 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.476687 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istio-data" pod "bfd5b4b9-3358-4549-80ba-5b84960fd3c0" (UID: "bfd5b4b9-3358-4549-80ba-5b84960fd3c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:41:06.477162 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.476989 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:41:06.477162 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.476994 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-credential-socket" pod "bfd5b4b9-3358-4549-80ba-5b84960fd3c0" (UID: "bfd5b4b9-3358-4549-80ba-5b84960fd3c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:41:06.477162 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.477009 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:41:06.477162 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.477024 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:41:06.477162 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.477040 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:41:06.477449 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.477247 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-workload-socket" pod "bfd5b4b9-3358-4549-80ba-5b84960fd3c0" (UID: "bfd5b4b9-3358-4549-80ba-5b84960fd3c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:41:06.477597 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.477553 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istiod-ca-cert" pod "b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" (UID: "b5e89f70-0dab-4ed5-b01b-2e78feab8b8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:41:06.477844 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.477797 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-credential-socket" pod "b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" (UID: "b5e89f70-0dab-4ed5-b01b-2e78feab8b8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:41:06.478061 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.478040 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-workload-certs" pod "b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" (UID: "b5e89f70-0dab-4ed5-b01b-2e78feab8b8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:41:06.478672 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.478394 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istiod-ca-cert" pod "bfd5b4b9-3358-4549-80ba-5b84960fd3c0" (UID: "bfd5b4b9-3358-4549-80ba-5b84960fd3c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:41:06.478672 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.478612 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istio-envoy" pod "bfd5b4b9-3358-4549-80ba-5b84960fd3c0" (UID: "bfd5b4b9-3358-4549-80ba-5b84960fd3c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:41:06.478842 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.478755 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-kube-api-access-zhdxt" pod "b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" (UID: "b5e89f70-0dab-4ed5-b01b-2e78feab8b8e"). InnerVolumeSpecName "kube-api-access-zhdxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:41:06.478842 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.478771 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istio-podinfo" pod "b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" (UID: "b5e89f70-0dab-4ed5-b01b-2e78feab8b8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:41:06.478997 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.478972 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-kube-api-access-xcxnd" pod "bfd5b4b9-3358-4549-80ba-5b84960fd3c0" (UID: "bfd5b4b9-3358-4549-80ba-5b84960fd3c0"). InnerVolumeSpecName "kube-api-access-xcxnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:41:06.479246 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.479223 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istio-podinfo" pod "bfd5b4b9-3358-4549-80ba-5b84960fd3c0" (UID: "bfd5b4b9-3358-4549-80ba-5b84960fd3c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:41:06.480289 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.480268 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istio-envoy" pod "b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" (UID: "b5e89f70-0dab-4ed5-b01b-2e78feab8b8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:41:06.480407 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.480387 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istio-token" pod "bfd5b4b9-3358-4549-80ba-5b84960fd3c0" (UID: "bfd5b4b9-3358-4549-80ba-5b84960fd3c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:41:06.480482 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.480454 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istio-token" pod "b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" (UID: "b5e89f70-0dab-4ed5-b01b-2e78feab8b8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:41:06.578103 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.578070 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:41:06.578103 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.578099 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:41:06.578103 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.578109 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:41:06.578352 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.578119 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:41:06.578352 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.578128 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:41:06.578352 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.578136 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:41:06.578352 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.578145 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:41:06.578352 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.578153 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:41:06.578352 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.578160 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:41:06.578352 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.578168 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:41:06.578352 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.578176 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcxnd\" (UniqueName: \"kubernetes.io/projected/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-kube-api-access-xcxnd\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:41:06.578352 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.578184 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:41:06.578352 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.578192 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhdxt\" (UniqueName: \"kubernetes.io/projected/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e-kube-api-access-zhdxt\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:41:06.578352 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.578200 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd5b4b9-3358-4549-80ba-5b84960fd3c0-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\"" Oct 30 00:41:06.845039 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.845004 5001 generic.go:356] "Generic (PLEG): container finished" podID="bfd5b4b9-3358-4549-80ba-5b84960fd3c0" containerID="1e2e0a73b6b9055d8b01356a1d7027e4571dd7d1d26adcb0f4cc8fb5ecf352a0" exitCode=0 Oct 30 00:41:06.845229 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.845065 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" Oct 30 00:41:06.845229 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.845081 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" event={"ID":"bfd5b4b9-3358-4549-80ba-5b84960fd3c0","Type":"ContainerDied","Data":"1e2e0a73b6b9055d8b01356a1d7027e4571dd7d1d26adcb0f4cc8fb5ecf352a0"} Oct 30 00:41:06.845229 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.845155 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl" event={"ID":"bfd5b4b9-3358-4549-80ba-5b84960fd3c0","Type":"ContainerDied","Data":"0feb8030ebec2f98b0429e42ae77a290dc5c6da9971c139247cbb8479cf6f6c5"} Oct 30 00:41:06.845229 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.845183 5001 scope.go:122] "RemoveContainer" containerID="1e2e0a73b6b9055d8b01356a1d7027e4571dd7d1d26adcb0f4cc8fb5ecf352a0" Oct 30 00:41:06.846555 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.846535 5001 generic.go:356] "Generic (PLEG): container finished" podID="b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" containerID="e42b653a6ed631241bde2d9d65f58d134b03524efc2d224f5e8766e5fff9044c" exitCode=0 Oct 30 00:41:06.846679 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.846598 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" event={"ID":"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e","Type":"ContainerDied","Data":"e42b653a6ed631241bde2d9d65f58d134b03524efc2d224f5e8766e5fff9044c"} Oct 30 00:41:06.846679 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.846644 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" event={"ID":"b5e89f70-0dab-4ed5-b01b-2e78feab8b8e","Type":"ContainerDied","Data":"c62e4a888861a39680d5b94a2b980e27ab10f267e055454e8f0bfe1b72e0787f"} Oct 30 00:41:06.846679 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.846603 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675" Oct 30 00:41:06.856554 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.856538 5001 scope.go:122] "RemoveContainer" containerID="1e2e0a73b6b9055d8b01356a1d7027e4571dd7d1d26adcb0f4cc8fb5ecf352a0" Oct 30 00:41:06.856817 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:41:06.856795 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e2e0a73b6b9055d8b01356a1d7027e4571dd7d1d26adcb0f4cc8fb5ecf352a0\": container with ID starting with 1e2e0a73b6b9055d8b01356a1d7027e4571dd7d1d26adcb0f4cc8fb5ecf352a0 not found: ID does not exist" containerID="1e2e0a73b6b9055d8b01356a1d7027e4571dd7d1d26adcb0f4cc8fb5ecf352a0" Oct 30 00:41:06.856894 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.856827 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e2e0a73b6b9055d8b01356a1d7027e4571dd7d1d26adcb0f4cc8fb5ecf352a0"} err="failed to get container status \"1e2e0a73b6b9055d8b01356a1d7027e4571dd7d1d26adcb0f4cc8fb5ecf352a0\": rpc error: code = NotFound desc = could not find container \"1e2e0a73b6b9055d8b01356a1d7027e4571dd7d1d26adcb0f4cc8fb5ecf352a0\": container with ID starting with 1e2e0a73b6b9055d8b01356a1d7027e4571dd7d1d26adcb0f4cc8fb5ecf352a0 not found: ID does not exist" Oct 30 00:41:06.856894 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.856850 5001 scope.go:122] "RemoveContainer" containerID="e42b653a6ed631241bde2d9d65f58d134b03524efc2d224f5e8766e5fff9044c" Oct 30 00:41:06.868701 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.868681 5001 scope.go:122] "RemoveContainer" containerID="e42b653a6ed631241bde2d9d65f58d134b03524efc2d224f5e8766e5fff9044c" Oct 30 00:41:06.869051 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:41:06.869024 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e42b653a6ed631241bde2d9d65f58d134b03524efc2d224f5e8766e5fff9044c\": container with ID starting with e42b653a6ed631241bde2d9d65f58d134b03524efc2d224f5e8766e5fff9044c not found: ID does not exist" containerID="e42b653a6ed631241bde2d9d65f58d134b03524efc2d224f5e8766e5fff9044c" Oct 30 00:41:06.869131 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.869061 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e42b653a6ed631241bde2d9d65f58d134b03524efc2d224f5e8766e5fff9044c"} err="failed to get container status \"e42b653a6ed631241bde2d9d65f58d134b03524efc2d224f5e8766e5fff9044c\": rpc error: code = NotFound desc = could not find container \"e42b653a6ed631241bde2d9d65f58d134b03524efc2d224f5e8766e5fff9044c\": container with ID starting with e42b653a6ed631241bde2d9d65f58d134b03524efc2d224f5e8766e5fff9044c not found: ID does not exist" Oct 30 00:41:06.890407 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.890385 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl"] Oct 30 00:41:06.900003 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.899980 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-m6zdl"] Oct 30 00:41:06.916748 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.916724 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfd5b4b9-3358-4549-80ba-5b84960fd3c0" path="/var/lib/kubelet/pods/bfd5b4b9-3358-4549-80ba-5b84960fd3c0/volumes" Oct 30 00:41:06.928539 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.928518 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675"] Oct 30 00:41:06.934858 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:06.934829 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6c45bb-openshift-default-5b89764557-7t675"] Oct 30 00:41:07.095161 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:07.095092 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-4n6qq_3febb7a0-47fa-4545-b76c-977789498353/manager/0.log" Oct 30 00:41:08.917104 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:08.917069 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" path="/var/lib/kubelet/pods/b5e89f70-0dab-4ed5-b01b-2e78feab8b8e/volumes" Oct 30 00:41:17.323526 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:41:17.323490 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:41:18.684064 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:18.684035 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-4n6qq_3febb7a0-47fa-4545-b76c-977789498353/manager/1.log" Oct 30 00:41:18.908279 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:18.908250 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-p564d_f408e652-4069-4d1d-b5b0-abc34593794c/manager/0.log" Oct 30 00:41:20.550452 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:20.550398 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-4n6qq_3febb7a0-47fa-4545-b76c-977789498353/manager/0.log" Oct 30 00:41:21.320141 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:21.320100 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-07a30c06-h4vkg_68744b2f-7594-448c-83f5-c2a3a7574274/revoke-keys/0.log" Oct 30 00:41:21.504731 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:21.504695 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-30713353-4nrqg_da718472-0862-4f25-8043-057c7fb59d17/revoke-keys/0.log" Oct 30 00:41:21.693042 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:21.692956 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-34e401f2-pl7wk_c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad/revoke-keys/0.log" Oct 30 00:41:21.878829 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:21.878798 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-545e045c-f9cf7_cd2803cf-8270-46c9-92ae-762022d4e340/revoke-keys/0.log" Oct 30 00:41:22.069877 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:22.069850 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-54b162fb-nrhl9_519641a7-422a-40ec-8150-084b8572ebc1/revoke-keys/0.log" Oct 30 00:41:22.254177 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:22.254147 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-778941da-sljf5_516e9889-a327-46ed-90f1-59189277128c/revoke-keys/0.log" Oct 30 00:41:22.442889 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:22.442814 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7e58d8ab-cxfng_0989dd93-7d85-4178-b9b0-48c87f0970a9/revoke-keys/0.log" Oct 30 00:41:22.623378 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:22.623348 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-85d234d0-r994g_f86ca908-fc4f-4c3a-a519-5d8f928576e3/revoke-keys/0.log" Oct 30 00:41:22.803946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:22.803897 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-91306600-8pxtv_ebb8b6c2-0ea1-4298-9bb8-162fa0e8fc8a/revoke-keys/0.log" Oct 30 00:41:22.988738 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:22.988709 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a307fa79-fmcgz_b4b18928-3805-413b-be9c-7732bc0072a9/revoke-keys/0.log" Oct 30 00:41:23.186518 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:23.186423 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-aa4e22fb-7kkcm_d51bbbf9-d605-49bb-ae09-426c7a5622c0/revoke-keys/0.log" Oct 30 00:41:23.373968 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:23.373940 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ab7e1fec-7vhxp_95d13539-bd4b-4108-ab25-d44712148107/revoke-keys/0.log" Oct 30 00:41:23.561712 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:23.561682 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c93bcf7b-d6xs9_f6cac301-2c7d-44e7-bad5-1b2a2c280dac/revoke-keys/0.log" Oct 30 00:41:23.750305 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:23.750272 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-cc72f0e2-k6fcc_2b1eb6e6-e4ce-44dc-b2ec-25a887349bd1/revoke-keys/0.log" Oct 30 00:41:23.934096 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:23.934013 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d87c9387-8v4s4_e0f9021e-ef29-4bda-9add-c386e7afd191/revoke-keys/0.log" Oct 30 00:41:24.131119 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:24.131089 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f2e7970f-mnckv_9780e16e-3caf-4dd7-bcba-6bcf4319ef1b/revoke-keys/0.log" Oct 30 00:41:24.324103 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:24.324071 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f5f7c4cb-rtqsc_14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078/revoke-keys/0.log" Oct 30 00:41:26.569213 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:26.569137 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-vxqwt_0aa4310d-ba64-4c3a-b1ca-0d5a8d1522fb/manager/0.log" Oct 30 00:41:26.757125 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:26.757096 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-z72x7_449bf72c-2ac6-42f7-8174-1e51e4c792cc/kuadrant-console-plugin/0.log" Oct 30 00:41:27.449781 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:27.449753 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-xr4lb_75b06ad5-f34b-4a66-b00a-50325ca9fc0b/manager/0.log" Oct 30 00:41:28.567087 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:28.567035 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd_8f5c5c45-659f-498f-8585-672fade7b41d/istio-proxy/0.log" Oct 30 00:41:28.761672 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:28.761641 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-2z4vm_d50b1c24-43df-4ca1-b869-5e1d0d9d67ce/discovery/1.log" Oct 30 00:41:29.909816 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:29.909774 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-ff75p_40f44089-5854-4325-820f-b3481352eb28/istio-proxy/0.log" Oct 30 00:41:30.279134 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:30.279107 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh_f9b032e2-dad9-4256-a988-8e6020b4da43/istio-proxy/0.log" Oct 30 00:41:30.800498 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:30.800468 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-645c8fd54c-hp29d_9d080b8e-cca6-4040-830e-cc9e9e25c50a/router/0.log" Oct 30 00:41:31.232011 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:31.231931 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-2z4vm_d50b1c24-43df-4ca1-b869-5e1d0d9d67ce/discovery/0.log" Oct 30 00:41:43.585043 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.585000 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hhxbv/must-gather-slrbp"] Oct 30 00:41:43.585902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.585548 5001 cpu_manager.go:414] "containerMap: removing container" podUID="516e9889-a327-46ed-90f1-59189277128c" containerName="revoke-keys" Oct 30 00:41:43.585902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.585567 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="516e9889-a327-46ed-90f1-59189277128c" containerName="revoke-keys" Oct 30 00:41:43.585902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.585586 5001 cpu_manager.go:414] "containerMap: removing container" podUID="bfd5b4b9-3358-4549-80ba-5b84960fd3c0" containerName="istio-proxy" Oct 30 00:41:43.585902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.585592 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfd5b4b9-3358-4549-80ba-5b84960fd3c0" containerName="istio-proxy" Oct 30 00:41:43.585902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.585603 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" containerName="istio-proxy" Oct 30 00:41:43.585902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.585610 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" containerName="istio-proxy" Oct 30 00:41:43.585902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.585620 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" Oct 30 00:41:43.585902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.585626 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" Oct 30 00:41:43.585902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.585652 5001 cpu_manager.go:414] "containerMap: removing container" podUID="63af7822-1fdf-4af9-977b-11ce91f5ec63" containerName="payload-pre-processing" Oct 30 00:41:43.585902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.585658 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63af7822-1fdf-4af9-977b-11ce91f5ec63" containerName="payload-pre-processing" Oct 30 00:41:43.585902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.585675 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e" containerName="payload-processing" Oct 30 00:41:43.585902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.585682 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e" containerName="payload-processing" Oct 30 00:41:43.585902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.585786 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="63af7822-1fdf-4af9-977b-11ce91f5ec63" containerName="payload-pre-processing" Oct 30 00:41:43.585902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.585800 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e81b3fe0-9090-4d0b-94cc-6d2ce82fde4e" containerName="payload-processing" Oct 30 00:41:43.585902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.585812 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="aadaf6fb-2702-4c04-88c9-cbb181049d68" containerName="istio-proxy" Oct 30 00:41:43.585902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.585822 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff8dd458-3a31-4084-b71e-2ecb4114c4e0" containerName="istio-proxy" Oct 30 00:41:43.585902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.585837 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="516e9889-a327-46ed-90f1-59189277128c" containerName="revoke-keys" Oct 30 00:41:43.585902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.585847 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfd5b4b9-3358-4549-80ba-5b84960fd3c0" containerName="istio-proxy" Oct 30 00:41:43.585902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.585855 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5e89f70-0dab-4ed5-b01b-2e78feab8b8e" containerName="istio-proxy" Oct 30 00:41:43.585902 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.585864 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0c7951f-8758-4b9c-b91a-00024f9bf582" containerName="main" Oct 30 00:41:43.589418 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.589396 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hhxbv/must-gather-slrbp" Oct 30 00:41:43.611055 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.611028 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hhxbv/must-gather-slrbp"] Oct 30 00:41:43.617480 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.617457 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hhxbv\"/\"openshift-service-ca.crt\"" Oct 30 00:41:43.617611 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.617514 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hhxbv\"/\"kube-root-ca.crt\"" Oct 30 00:41:43.617611 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.617564 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hhxbv\"/\"default-dockercfg-6cngd\"" Oct 30 00:41:43.681662 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.681627 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pr5qw/must-gather-4jsrj"] Oct 30 00:41:43.687675 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.687642 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pr5qw/must-gather-4jsrj" Oct 30 00:41:43.705673 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.705644 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pr5qw/must-gather-4jsrj"] Oct 30 00:41:43.706415 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.706390 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0c2c2491-6de8-40fe-8ff3-4e0f4e5e5714-must-gather-output\") pod \"must-gather-slrbp\" (UID: \"0c2c2491-6de8-40fe-8ff3-4e0f4e5e5714\") " pod="openshift-must-gather-hhxbv/must-gather-slrbp" Oct 30 00:41:43.706554 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.706533 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwpxf\" (UniqueName: \"kubernetes.io/projected/0c2c2491-6de8-40fe-8ff3-4e0f4e5e5714-kube-api-access-hwpxf\") pod \"must-gather-slrbp\" (UID: \"0c2c2491-6de8-40fe-8ff3-4e0f4e5e5714\") " pod="openshift-must-gather-hhxbv/must-gather-slrbp" Oct 30 00:41:43.708100 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.708077 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pr5qw\"/\"openshift-service-ca.crt\"" Oct 30 00:41:43.708224 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.708099 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pr5qw\"/\"kube-root-ca.crt\"" Oct 30 00:41:43.708224 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.708143 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pr5qw\"/\"default-dockercfg-ms8r5\"" Oct 30 00:41:43.807477 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.807441 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e9eafbbb-4c19-4d93-bc1f-52571530ede7-must-gather-output\") pod \"must-gather-4jsrj\" (UID: \"e9eafbbb-4c19-4d93-bc1f-52571530ede7\") " pod="openshift-must-gather-pr5qw/must-gather-4jsrj" Oct 30 00:41:43.807659 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.807561 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0c2c2491-6de8-40fe-8ff3-4e0f4e5e5714-must-gather-output\") pod \"must-gather-slrbp\" (UID: \"0c2c2491-6de8-40fe-8ff3-4e0f4e5e5714\") " pod="openshift-must-gather-hhxbv/must-gather-slrbp" Oct 30 00:41:43.807659 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.807640 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4b54\" (UniqueName: \"kubernetes.io/projected/e9eafbbb-4c19-4d93-bc1f-52571530ede7-kube-api-access-j4b54\") pod \"must-gather-4jsrj\" (UID: \"e9eafbbb-4c19-4d93-bc1f-52571530ede7\") " pod="openshift-must-gather-pr5qw/must-gather-4jsrj" Oct 30 00:41:43.807743 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.807660 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwpxf\" (UniqueName: \"kubernetes.io/projected/0c2c2491-6de8-40fe-8ff3-4e0f4e5e5714-kube-api-access-hwpxf\") pod \"must-gather-slrbp\" (UID: \"0c2c2491-6de8-40fe-8ff3-4e0f4e5e5714\") " pod="openshift-must-gather-hhxbv/must-gather-slrbp" Oct 30 00:41:43.808029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.808009 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0c2c2491-6de8-40fe-8ff3-4e0f4e5e5714-must-gather-output\") pod \"must-gather-slrbp\" (UID: \"0c2c2491-6de8-40fe-8ff3-4e0f4e5e5714\") " pod="openshift-must-gather-hhxbv/must-gather-slrbp" Oct 30 00:41:43.830735 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.830706 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwpxf\" (UniqueName: \"kubernetes.io/projected/0c2c2491-6de8-40fe-8ff3-4e0f4e5e5714-kube-api-access-hwpxf\") pod \"must-gather-slrbp\" (UID: \"0c2c2491-6de8-40fe-8ff3-4e0f4e5e5714\") " pod="openshift-must-gather-hhxbv/must-gather-slrbp" Oct 30 00:41:43.904777 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.904704 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hhxbv/must-gather-slrbp" Oct 30 00:41:43.908929 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.908890 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4b54\" (UniqueName: \"kubernetes.io/projected/e9eafbbb-4c19-4d93-bc1f-52571530ede7-kube-api-access-j4b54\") pod \"must-gather-4jsrj\" (UID: \"e9eafbbb-4c19-4d93-bc1f-52571530ede7\") " pod="openshift-must-gather-pr5qw/must-gather-4jsrj" Oct 30 00:41:43.909048 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.908986 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e9eafbbb-4c19-4d93-bc1f-52571530ede7-must-gather-output\") pod \"must-gather-4jsrj\" (UID: \"e9eafbbb-4c19-4d93-bc1f-52571530ede7\") " pod="openshift-must-gather-pr5qw/must-gather-4jsrj" Oct 30 00:41:43.909310 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.909290 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e9eafbbb-4c19-4d93-bc1f-52571530ede7-must-gather-output\") pod \"must-gather-4jsrj\" (UID: \"e9eafbbb-4c19-4d93-bc1f-52571530ede7\") " pod="openshift-must-gather-pr5qw/must-gather-4jsrj" Oct 30 00:41:43.929789 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:43.929768 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4b54\" (UniqueName: \"kubernetes.io/projected/e9eafbbb-4c19-4d93-bc1f-52571530ede7-kube-api-access-j4b54\") pod \"must-gather-4jsrj\" (UID: \"e9eafbbb-4c19-4d93-bc1f-52571530ede7\") " pod="openshift-must-gather-pr5qw/must-gather-4jsrj" Oct 30 00:41:44.004391 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:44.004362 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pr5qw/must-gather-4jsrj" Oct 30 00:41:44.203133 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:44.203100 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hhxbv/must-gather-slrbp"] Oct 30 00:41:44.204040 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:41:44.204014 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c2c2491_6de8_40fe_8ff3_4e0f4e5e5714.slice/crio-a5f48b6ef2fb93c3d9d3766a4d570e8c38e4172ab2bd9ac326e70ac83972b3b3 WatchSource:0}: Error finding container a5f48b6ef2fb93c3d9d3766a4d570e8c38e4172ab2bd9ac326e70ac83972b3b3: Status 404 returned error can't find the container with id a5f48b6ef2fb93c3d9d3766a4d570e8c38e4172ab2bd9ac326e70ac83972b3b3 Oct 30 00:41:44.205782 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:44.205765 5001 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:41:44.304750 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:44.304717 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pr5qw/must-gather-4jsrj"] Oct 30 00:41:44.305463 ip-10-0-143-120 kubenswrapper[5001]: W1030 00:41:44.305440 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9eafbbb_4c19_4d93_bc1f_52571530ede7.slice/crio-1b0448759eb5565c4e3a4cf16cdd073c845860ef4b03a8e7acee8b092e09b8fd WatchSource:0}: Error finding container 1b0448759eb5565c4e3a4cf16cdd073c845860ef4b03a8e7acee8b092e09b8fd: Status 404 returned error can't find the container with id 1b0448759eb5565c4e3a4cf16cdd073c845860ef4b03a8e7acee8b092e09b8fd Oct 30 00:41:45.107723 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:45.107685 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hhxbv/must-gather-slrbp" event={"ID":"0c2c2491-6de8-40fe-8ff3-4e0f4e5e5714","Type":"ContainerStarted","Data":"a5f48b6ef2fb93c3d9d3766a4d570e8c38e4172ab2bd9ac326e70ac83972b3b3"} Oct 30 00:41:45.108964 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:45.108937 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pr5qw/must-gather-4jsrj" event={"ID":"e9eafbbb-4c19-4d93-bc1f-52571530ede7","Type":"ContainerStarted","Data":"1b0448759eb5565c4e3a4cf16cdd073c845860ef4b03a8e7acee8b092e09b8fd"} Oct 30 00:41:46.136519 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:46.136474 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pr5qw/must-gather-4jsrj" event={"ID":"e9eafbbb-4c19-4d93-bc1f-52571530ede7","Type":"ContainerStarted","Data":"d66f353bf96972b6bd805713759e3e5db0ffd78b06ccd70a6fa98ca1da5db6d5"} Oct 30 00:41:46.137029 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:46.136538 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pr5qw/must-gather-4jsrj" event={"ID":"e9eafbbb-4c19-4d93-bc1f-52571530ede7","Type":"ContainerStarted","Data":"8cb2659cb61d8e6b3bd590ade0790e77f126e4d5820e9a14c4b793e34ccb20a2"} Oct 30 00:41:46.182250 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:46.182186 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-pr5qw/must-gather-4jsrj" podStartSLOduration=2.322703313 podStartE2EDuration="3.182164912s" totalImagesPullingTime="859.461599ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:46.171280418 +0000 UTC m=+2221.376781601" watchObservedRunningTime="2026-10-30 00:41:46.182164912 +0000 UTC m=+2221.387666093" Oct 30 00:41:47.332448 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:41:47.332406 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:41:49.142837 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:49.142805 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zm28x_fb8917db-c1ba-43af-995e-1948de69aa29/global-pull-secret-syncer/0.log" Oct 30 00:41:49.321739 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:49.321703 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-4fpvt_c3ce5e93-7d27-4fa6-b360-e571d8f18481/connection-checker/0.log" Oct 30 00:41:49.448316 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:49.448240 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-b9fcp_6f2b3396-a95d-4bc7-902f-1daf3dd689fd/connection-checker/0.log" Oct 30 00:41:49.570411 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:49.570383 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-m98x7_d2413398-197b-4cad-8dc4-9bba9effe3b7/connection-checker/0.log" Oct 30 00:41:50.011727 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:50.001233 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-q8m2x_5166c4d7-2b82-4bbd-8d58-836cb274ed25/konnectivity-agent/0.log" Oct 30 00:41:50.051335 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:50.051304 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54b162fb-nrhl9"] Oct 30 00:41:50.056014 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:50.055976 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54b162fb-nrhl9"] Oct 30 00:41:50.394300 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:50.390945 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-120.ec2.internal_4dc21da3207f1de3351a28354872ea48/haproxy/0.log" Oct 30 00:41:50.949932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:50.945050 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="519641a7-422a-40ec-8150-084b8572ebc1" path="/var/lib/kubelet/pods/519641a7-422a-40ec-8150-084b8572ebc1/volumes" Oct 30 00:41:51.378933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:51.373376 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-4n6qq_3febb7a0-47fa-4545-b76c-977789498353/manager/0.log" Oct 30 00:41:51.398651 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:51.386995 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-4n6qq_3febb7a0-47fa-4545-b76c-977789498353/manager/1.log" Oct 30 00:41:52.694679 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:52.694641 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-p564d_f408e652-4069-4d1d-b5b0-abc34593794c/manager/0.log" Oct 30 00:41:55.453515 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:55.453473 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-4n6qq_3febb7a0-47fa-4545-b76c-977789498353/manager/0.log" Oct 30 00:41:55.455548 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:55.455523 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-4n6qq_3febb7a0-47fa-4545-b76c-977789498353/manager/1.log" Oct 30 00:41:56.228889 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:56.228659 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-p564d_f408e652-4069-4d1d-b5b0-abc34593794c/manager/0.log" Oct 30 00:41:56.351108 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:56.349933 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hhxbv/must-gather-slrbp" event={"ID":"0c2c2491-6de8-40fe-8ff3-4e0f4e5e5714","Type":"ContainerStarted","Data":"9c0132c2fd50e0640790559ca63a5c4f0cc9b516f86fadf0cbbe1c6d7702c454"} Oct 30 00:41:56.351108 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:56.349995 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hhxbv/must-gather-slrbp" event={"ID":"0c2c2491-6de8-40fe-8ff3-4e0f4e5e5714","Type":"ContainerStarted","Data":"55895db38841d58634b93d1ae680bb405a85afab8d57e2ee21c9bc191c089cc2"} Oct 30 00:41:56.400932 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:56.396320 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-hhxbv/must-gather-slrbp" podStartSLOduration=2.268988666 podStartE2EDuration="13.396298916s" totalImagesPullingTime="11.12731025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:56.394323133 +0000 UTC m=+2231.599824314" watchObservedRunningTime="2026-10-30 00:41:56.396298916 +0000 UTC m=+2231.601800097" Oct 30 00:41:57.207291 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:57.207247 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-vxqwt_0aa4310d-ba64-4c3a-b1ca-0d5a8d1522fb/manager/0.log" Oct 30 00:41:57.332934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:57.332281 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-z72x7_449bf72c-2ac6-42f7-8174-1e51e4c792cc/kuadrant-console-plugin/0.log" Oct 30 00:41:58.078937 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:58.078755 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-xr4lb_75b06ad5-f34b-4a66-b00a-50325ca9fc0b/manager/0.log" Oct 30 00:41:58.603365 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:58.603294 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-4n6qq_3febb7a0-47fa-4545-b76c-977789498353/manager/0.log" Oct 30 00:41:58.608690 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:58.608663 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-4n6qq_3febb7a0-47fa-4545-b76c-977789498353/manager/1.log" Oct 30 00:41:59.167936 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:59.167856 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-4n6qq_3febb7a0-47fa-4545-b76c-977789498353/manager/0.log" Oct 30 00:41:59.189933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:59.187862 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-4n6qq_3febb7a0-47fa-4545-b76c-977789498353/manager/1.log" Oct 30 00:41:59.246931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:59.246104 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-p564d_f408e652-4069-4d1d-b5b0-abc34593794c/manager/0.log" Oct 30 00:41:59.791770 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:41:59.790931 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-p564d_f408e652-4069-4d1d-b5b0-abc34593794c/manager/0.log" Oct 30 00:42:00.028773 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:00.028726 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d87c9387-8v4s4"] Oct 30 00:42:00.036961 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:00.036894 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d87c9387-8v4s4"] Oct 30 00:42:00.121988 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:00.121873 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pr5qw/perf-node-gather-daemonset-snd2b"] Oct 30 00:42:00.126726 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:00.126700 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-snd2b" Oct 30 00:42:00.159933 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:00.153095 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pr5qw/perf-node-gather-daemonset-snd2b"] Oct 30 00:42:00.280693 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:00.280653 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f41d0d1-1114-4118-8146-8bca365b0b28-sys\") pod \"perf-node-gather-daemonset-snd2b\" (UID: \"3f41d0d1-1114-4118-8146-8bca365b0b28\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-snd2b" Oct 30 00:42:00.280885 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:00.280765 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s4hr\" (UniqueName: \"kubernetes.io/projected/3f41d0d1-1114-4118-8146-8bca365b0b28-kube-api-access-5s4hr\") pod \"perf-node-gather-daemonset-snd2b\" (UID: \"3f41d0d1-1114-4118-8146-8bca365b0b28\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-snd2b" Oct 30 00:42:00.280885 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:00.280830 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3f41d0d1-1114-4118-8146-8bca365b0b28-podres\") pod \"perf-node-gather-daemonset-snd2b\" (UID: \"3f41d0d1-1114-4118-8146-8bca365b0b28\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-snd2b" Oct 30 00:42:00.280999 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:00.280953 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3f41d0d1-1114-4118-8146-8bca365b0b28-lib-modules\") pod \"perf-node-gather-daemonset-snd2b\" (UID: \"3f41d0d1-1114-4118-8146-8bca365b0b28\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-snd2b" Oct 30 00:42:00.281045 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:00.281013 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3f41d0d1-1114-4118-8146-8bca365b0b28-proc\") pod \"perf-node-gather-daemonset-snd2b\" (UID: \"3f41d0d1-1114-4118-8146-8bca365b0b28\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-snd2b" Oct 30 00:42:00.382560 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:00.382466 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f41d0d1-1114-4118-8146-8bca365b0b28-sys\") pod \"perf-node-gather-daemonset-snd2b\" (UID: \"3f41d0d1-1114-4118-8146-8bca365b0b28\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-snd2b" Oct 30 00:42:00.382560 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:00.382519 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s4hr\" (UniqueName: \"kubernetes.io/projected/3f41d0d1-1114-4118-8146-8bca365b0b28-kube-api-access-5s4hr\") pod \"perf-node-gather-daemonset-snd2b\" (UID: \"3f41d0d1-1114-4118-8146-8bca365b0b28\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-snd2b" Oct 30 00:42:00.382816 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:00.382566 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3f41d0d1-1114-4118-8146-8bca365b0b28-podres\") pod \"perf-node-gather-daemonset-snd2b\" (UID: \"3f41d0d1-1114-4118-8146-8bca365b0b28\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-snd2b" Oct 30 00:42:00.382816 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:00.382605 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3f41d0d1-1114-4118-8146-8bca365b0b28-lib-modules\") pod \"perf-node-gather-daemonset-snd2b\" (UID: \"3f41d0d1-1114-4118-8146-8bca365b0b28\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-snd2b" Oct 30 00:42:00.382816 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:00.382603 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f41d0d1-1114-4118-8146-8bca365b0b28-sys\") pod \"perf-node-gather-daemonset-snd2b\" (UID: \"3f41d0d1-1114-4118-8146-8bca365b0b28\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-snd2b" Oct 30 00:42:00.382816 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:00.382657 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3f41d0d1-1114-4118-8146-8bca365b0b28-podres\") pod \"perf-node-gather-daemonset-snd2b\" (UID: \"3f41d0d1-1114-4118-8146-8bca365b0b28\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-snd2b" Oct 30 00:42:00.382816 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:00.382683 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3f41d0d1-1114-4118-8146-8bca365b0b28-proc\") pod \"perf-node-gather-daemonset-snd2b\" (UID: \"3f41d0d1-1114-4118-8146-8bca365b0b28\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-snd2b" Oct 30 00:42:00.382816 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:00.382719 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3f41d0d1-1114-4118-8146-8bca365b0b28-lib-modules\") pod \"perf-node-gather-daemonset-snd2b\" (UID: \"3f41d0d1-1114-4118-8146-8bca365b0b28\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-snd2b" Oct 30 00:42:00.382816 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:00.382788 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3f41d0d1-1114-4118-8146-8bca365b0b28-proc\") pod \"perf-node-gather-daemonset-snd2b\" (UID: \"3f41d0d1-1114-4118-8146-8bca365b0b28\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-snd2b" Oct 30 00:42:00.403796 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:00.403762 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s4hr\" (UniqueName: \"kubernetes.io/projected/3f41d0d1-1114-4118-8146-8bca365b0b28-kube-api-access-5s4hr\") pod \"perf-node-gather-daemonset-snd2b\" (UID: \"3f41d0d1-1114-4118-8146-8bca365b0b28\") " pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-snd2b" Oct 30 00:42:00.443496 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:00.443452 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-snd2b" Oct 30 00:42:00.628900 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:00.628870 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-nqw98_aff3a44b-6cad-449e-8733-f85e9899ecc3/cluster-monitoring-operator/0.log" Oct 30 00:42:00.754087 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:00.754035 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pr5qw/perf-node-gather-daemonset-snd2b"] Oct 30 00:42:00.922326 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:00.922301 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0f9021e-ef29-4bda-9add-c386e7afd191" path="/var/lib/kubelet/pods/e0f9021e-ef29-4bda-9add-c386e7afd191/volumes" Oct 30 00:42:01.347851 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:01.347821 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qdn9n_a1b58bee-c614-4dbf-bf3c-3593b3da5069/node-exporter/0.log" Oct 30 00:42:01.437200 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:01.437154 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-snd2b" event={"ID":"3f41d0d1-1114-4118-8146-8bca365b0b28","Type":"ContainerStarted","Data":"ed899ef1cb3980aed140892cdb518e782b8643dae3bfdcec20583034dfb8eeec"} Oct 30 00:42:01.437398 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:01.437219 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-snd2b" event={"ID":"3f41d0d1-1114-4118-8146-8bca365b0b28","Type":"ContainerStarted","Data":"6f0243e2c05beb971b032dd14c075b4d317a6da2e4301218ed2ca2115f30a665"} Oct 30 00:42:01.437398 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:01.437278 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-snd2b" Oct 30 00:42:01.472946 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:01.470102 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qdn9n_a1b58bee-c614-4dbf-bf3c-3593b3da5069/kube-rbac-proxy/0.log" Oct 30 00:42:01.478775 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:01.478304 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-snd2b" podStartSLOduration=1.478283265 podStartE2EDuration="1.478283265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:01.473560548 +0000 UTC m=+2236.679061759" watchObservedRunningTime="2026-10-30 00:42:01.478283265 +0000 UTC m=+2236.683784450" Oct 30 00:42:01.591586 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:01.591550 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qdn9n_a1b58bee-c614-4dbf-bf3c-3593b3da5069/init-textfile/0.log" Oct 30 00:42:02.305965 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:02.305876 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-4n6qq_3febb7a0-47fa-4545-b76c-977789498353/manager/0.log" Oct 30 00:42:02.309019 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:02.308988 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-4n6qq_3febb7a0-47fa-4545-b76c-977789498353/manager/1.log" Oct 30 00:42:02.827517 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:02.827478 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd_8f5c5c45-659f-498f-8585-672fade7b41d/istio-proxy/0.log" Oct 30 00:42:02.883307 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:02.883273 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-p564d_f408e652-4069-4d1d-b5b0-abc34593794c/manager/0.log" Oct 30 00:42:02.921288 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:02.921254 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-2z4vm_d50b1c24-43df-4ca1-b869-5e1d0d9d67ce/discovery/1.log" Oct 30 00:42:04.918602 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:04.918567 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-2z4vm_d50b1c24-43df-4ca1-b869-5e1d0d9d67ce/discovery/0.log" Oct 30 00:42:06.088751 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:06.088716 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-4n6qq_3febb7a0-47fa-4545-b76c-977789498353/manager/0.log" Oct 30 00:42:06.091429 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:06.091400 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-4n6qq_3febb7a0-47fa-4545-b76c-977789498353/manager/1.log" Oct 30 00:42:06.652777 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:06.652738 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-p564d_f408e652-4069-4d1d-b5b0-abc34593794c/manager/0.log" Oct 30 00:42:06.727587 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:06.727558 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-7m2x2_0d3be8a7-773b-42e1-a9a1-60d7df717301/networking-console-plugin/0.log" Oct 30 00:42:07.187744 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:07.187707 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-ff75p_40f44089-5854-4325-820f-b3481352eb28/istio-proxy/0.log" Oct 30 00:42:07.314428 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:07.314396 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh_f9b032e2-dad9-4256-a988-8e6020b4da43/istio-proxy/0.log" Oct 30 00:42:07.459365 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:07.459256 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pr5qw/perf-node-gather-daemonset-snd2b" Oct 30 00:42:07.483161 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:07.483131 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-645c8fd54c-hp29d_9d080b8e-cca6-4040-830e-cc9e9e25c50a/router/0.log" Oct 30 00:42:07.858235 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:07.858203 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8gb5_9fa688d8-29ca-4c1f-926a-f1a4099e6b44/console-operator/2.log" Oct 30 00:42:07.862870 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:07.862837 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8gb5_9fa688d8-29ca-4c1f-926a-f1a4099e6b44/console-operator/3.log" Oct 30 00:42:09.923931 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:09.920266 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd_8f5c5c45-659f-498f-8585-672fade7b41d/istio-proxy/0.log" Oct 30 00:42:10.029212 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:10.029174 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-2z4vm_d50b1c24-43df-4ca1-b869-5e1d0d9d67ce/discovery/1.log" Oct 30 00:42:10.278373 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:10.278335 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-n78wz_4350294b-64f3-4efd-9619-90c013d82432/volume-data-source-validator/0.log" Oct 30 00:42:11.059199 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:11.059167 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-vxqwt_0aa4310d-ba64-4c3a-b1ca-0d5a8d1522fb/manager/0.log" Oct 30 00:42:11.121654 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:11.121605 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-z72x7_449bf72c-2ac6-42f7-8174-1e51e4c792cc/kuadrant-console-plugin/0.log" Oct 30 00:42:11.508864 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:11.508707 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-xr4lb_75b06ad5-f34b-4a66-b00a-50325ca9fc0b/manager/0.log" Oct 30 00:42:12.173752 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:12.173694 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-2z4vm_d50b1c24-43df-4ca1-b869-5e1d0d9d67ce/discovery/0.log" Oct 30 00:42:12.982662 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:12.982617 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-52kw4_acc82f37-fad2-424d-89d0-5e5876cb75de/dns/0.log" Oct 30 00:42:13.100247 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:13.100214 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-52kw4_acc82f37-fad2-424d-89d0-5e5876cb75de/kube-rbac-proxy/0.log" Oct 30 00:42:13.832710 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:13.832678 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b54v5_a9e7a0c8-43b3-4f8c-8870-505edace7f78/dns-node-resolver/0.log" Oct 30 00:42:14.354900 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:14.354869 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd_8f5c5c45-659f-498f-8585-672fade7b41d/istio-proxy/0.log" Oct 30 00:42:14.367871 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:14.367843 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-ff75p_40f44089-5854-4325-820f-b3481352eb28/istio-proxy/0.log" Oct 30 00:42:14.451253 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:14.451222 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-2z4vm_d50b1c24-43df-4ca1-b869-5e1d0d9d67ce/discovery/1.log" Oct 30 00:42:14.493256 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:14.493214 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh_f9b032e2-dad9-4256-a988-8e6020b4da43/istio-proxy/0.log" Oct 30 00:42:14.683848 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:14.683765 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-645c8fd54c-hp29d_9d080b8e-cca6-4040-830e-cc9e9e25c50a/router/0.log" Oct 30 00:42:15.708023 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:15.707984 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qs54p_85b08631-ff08-461f-8bda-6a64a609e1f1/node-ca/0.log" Oct 30 00:42:16.555846 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:16.555807 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-2z4vm_d50b1c24-43df-4ca1-b869-5e1d0d9d67ce/discovery/0.log" Oct 30 00:42:17.028545 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:17.028506 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34e401f2-pl7wk"] Oct 30 00:42:17.038217 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:17.038189 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34e401f2-pl7wk"] Oct 30 00:42:17.331296 ip-10-0-143-120 kubenswrapper[5001]: E1030 00:42:17.331160 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dd8wx" containerName="extractor" Oct 30 00:42:17.895399 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:17.895369 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68dd7b898f9v5fd_8f5c5c45-659f-498f-8585-672fade7b41d/istio-proxy/0.log" Oct 30 00:42:18.027793 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:18.027755 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5f7c4cb-rtqsc"] Oct 30 00:42:18.039508 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:18.039454 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5f7c4cb-rtqsc"] Oct 30 00:42:18.062827 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:18.062788 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-2z4vm_d50b1c24-43df-4ca1-b869-5e1d0d9d67ce/discovery/1.log" Oct 30 00:42:18.755934 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:18.754153 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-ff75p_40f44089-5854-4325-820f-b3481352eb28/istio-proxy/0.log" Oct 30 00:42:18.874146 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:18.874112 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6bd8768855-n4bgh_f9b032e2-dad9-4256-a988-8e6020b4da43/istio-proxy/0.log" Oct 30 00:42:18.923270 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:18.923234 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078" path="/var/lib/kubelet/pods/14d68ed9-c3b2-48f4-8edb-0d4c2c5f0078/volumes" Oct 30 00:42:18.923858 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:18.923834 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad" path="/var/lib/kubelet/pods/c3efe694-3bc9-4e6e-9e3a-99ffd0dd7bad/volumes" Oct 30 00:42:19.088000 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:19.087960 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-645c8fd54c-hp29d_9d080b8e-cca6-4040-830e-cc9e9e25c50a/router/0.log" Oct 30 00:42:19.238617 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:19.237746 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-vxqwt_0aa4310d-ba64-4c3a-b1ca-0d5a8d1522fb/manager/0.log" Oct 30 00:42:19.295543 ip-10-0-143-120 kubenswrapper[5001]: I1030 00:42:19.295507 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-z72x7_449bf72c-2ac6-42f7-8174-1e51e4c792cc/kuadrant-console-plugin/0.log"